Nov 02 19:45:02.004459 ip-10-0-134-196 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 19:45:02.004470 ip-10-0-134-196 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 19:45:02.004477 ip-10-0-134-196 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 19:45:02.004697 ip-10-0-134-196 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 19:45:12.209316 ip-10-0-134-196 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 19:45:12.209332 ip-10-0-134-196 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4568f2d9d30140c7a35aea1d6b8b206b --
Nov 02 19:47:25.383059 ip-10-0-134-196 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 19:47:25.772349 ip-10-0-134-196 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:47:25.772349 ip-10-0-134-196 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 19:47:25.772349 ip-10-0-134-196 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:47:25.772349 ip-10-0-134-196 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:47:25.775598 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775519    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:47:25.775598 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775596    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:47:25.775598 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775601    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775604    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775607    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775613    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775617    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775620    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775623    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775626    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775628    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775631    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775634    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775637    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775640    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775642    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775645    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775648    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775650    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775653    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775656    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:47:25.775687 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775658    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775661    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775664    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775667    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775669    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775672    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775676    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775679    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775682    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775684    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775687    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775690    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775693    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775696    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775699    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775701    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775704    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775707    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775709    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:47:25.776152 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775712    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775714    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775717    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775720    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775722    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775725    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775727    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775730    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775733    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775735    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775738    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775741    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775743    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775746    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775748    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775754    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775758    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775761    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775776    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775780    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:47:25.776631 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775784    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775787    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775790    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775793    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775795    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775798    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775801    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775803    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775807    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775810    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775813    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775816    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775818    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775821    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775823    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775826    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775829    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775831    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775834    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775837    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:47:25.777110 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775839    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775842    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775844    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775847    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775850    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775853    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775856    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775859    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775862    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775865    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775868    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775871    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775874    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775877    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775880    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775883    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775885    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775888    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775891    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:47:25.777577 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775894    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775896    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775899    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775903    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775906    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775909    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.775911    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776699    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776705    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776708    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776711    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776714    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776717    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776720    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776723    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776725    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776728    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776731    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776735    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776737    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:47:25.778045 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776741    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776744    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776747    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776749    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776752    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776755    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776758    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776761    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776778    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776783    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776787    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776792    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776796    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776799    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776802    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776805    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776808    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776811    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776814    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776817    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:47:25.778522 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776819    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776822    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776825    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776827    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776830    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776834    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776837    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776840    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776843    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776846    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776848    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776851    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776854    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776856    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776859    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776862    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776864    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776867    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776869    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:47:25.779015 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776875    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776879    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776882    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776884    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776887    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776889    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776892    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776895    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776897    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776902    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776906    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776909    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776912    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776915    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776918    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776921    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776924    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776926    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776930    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:47:25.779498 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776932    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776935    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776938    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776941    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776943    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776946    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776949    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776951    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776954    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776957    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776959    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776962    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776965    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776969    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776972    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776975    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776978    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776981    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776983    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:47:25.779960 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776986    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776989    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776992    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776994    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776997    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.776999    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777002    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777005    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777007    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777010    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777012    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777015    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777018    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777020    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777023    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777026    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777506    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777511    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777514    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777516    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:47:25.780424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777519    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777521    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777524    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777527    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777529    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777532    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777536    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777539    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777542    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777545    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777548    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777550    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777553    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777555    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777558    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777561    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777563    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777566    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777569    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:47:25.780899 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777572    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777575    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777577    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777580    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777583    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777586    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777589    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777592    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777594    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777597    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777600    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777602    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777605    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777607    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777610    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777612    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777615    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777618    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777620    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777623    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:47:25.781343 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777626    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777629    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777631    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777634    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777636    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777639    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777641    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777644    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777647    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777649    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777652    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777655    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777657    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777660    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777663    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777665    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777668    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777670    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777673    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777676    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:47:25.781826 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777678    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777681    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777683    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777686    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777688    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777691    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777693    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777697    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777702    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777705    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777708    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777712    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777715    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777718    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777720    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777724    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777727    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777730    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777733    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:47:25.782288 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777735    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777738    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777741    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777743    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777747    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777749    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777752    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777755    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777757    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777760    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777762    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777779    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777781    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777784    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777787    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777790    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777792    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777795    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777797    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777800    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:47:25.782745 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777802    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777807    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777812    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.777816    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778500    2578 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778509    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778516    2578 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778521    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778525    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778529    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778535    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778539    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778543    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778546    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778550    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778553    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778557    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778560    2578 flags.go:64] FLAG: --cgroup-root=""
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778563    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778566    2578 flags.go:64] FLAG: --client-ca-file=""
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778569    2578 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778572    2578 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778576    2578 flags.go:64] FLAG: --cluster-domain=""
Nov 02 19:47:25.783224 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778579    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778582    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778585    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778589    2578 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778593    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778596    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778599    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778603    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778606    2578 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778609    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778612    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778616    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778619    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778624    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778627    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778630    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778633    2578 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778636    2578 flags.go:64] FLAG: --enable-server="true"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778639    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778648    2578 flags.go:64] FLAG: --event-burst="100"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778651    2578 flags.go:64] FLAG: --event-qps="50"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778654    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778658    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778661    2578 flags.go:64] FLAG: --eviction-hard=""
Nov 02 19:47:25.783746 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778670    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778673    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778676    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778679    2578 flags.go:64] FLAG: --eviction-soft=""
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778682    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778685    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778688    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778691    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778694    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778697    2578 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778700    2578 flags.go:64] FLAG: --feature-gates=""
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778704    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778707    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778710    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778713    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778716    2578 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778719    2578 flags.go:64] FLAG: --help="false"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778722    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778726    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778729    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778734    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778737    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778741    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 19:47:25.784351 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778744    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778747    2578 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778750    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778754    2578 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778757    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778761    2578 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778776    2578 flags.go:64] FLAG: --kube-reserved=""
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778780    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778783    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778786    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778789    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778792    2578 flags.go:64] FLAG: --lock-file=""
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778795    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778798    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778801    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778807    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778810    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778813    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778816    2578 flags.go:64] FLAG: --logging-format="text"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778819    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778823    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778826    2578 flags.go:64] FLAG: --manifest-url=""
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778829    2578 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778834    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778837    2578 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 19:47:25.785423 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778841    2578 flags.go:64] FLAG: --max-pods="110"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778844    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778847    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778850    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778855    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778858    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778861    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778864    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778871    2578 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778875    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778878    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778881    2578 flags.go:64] FLAG: --pod-cidr=""
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778884    2578 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778888    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778891    2578 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778894    2578 flags.go:64] FLAG: --port="10250"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778897    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778900    2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-02e70aca871c76228"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778903    2578 flags.go:64] FLAG: --qos-reserved=""
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778906    2578 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778909    2578 flags.go:64] FLAG: --register-node="true"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778913    2578 flags.go:64] FLAG: --register-with-taints=""
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778917    2578 flags.go:64] FLAG: --registry-burst="10"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778920    2578 flags.go:64] FLAG: --registry-qps="5"
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778922    2578 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778925    2578 flags.go:64] FLAG: --reserved-memory=""
Nov 02 19:47:25.786322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778929    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778933    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778936    2578 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778939    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778941    2578 flags.go:64] FLAG: --runonce="false"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778944    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778948    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778951    2578 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778954    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778957    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778962    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778965    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778968    2578 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778971    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778974    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778977    2578 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778980    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778983    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778986    2578 flags.go:64] FLAG: --system-cgroups=""
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778989    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778995    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.778999    2578 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.779002    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.779005    2578 flags.go:64] FLAG: --tls-min-version=""
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.779008    2578 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 19:47:25.786971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.779011    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.779014    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.779017    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.779021    2578 flags.go:64] FLAG: --v="2"
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.779025    2578 flags.go:64] FLAG: --version="false"
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.779030    2578 flags.go:64] FLAG: --vmodule=""
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.779034    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.779037    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779174    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779177    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779181    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779184    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779187    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779190    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779192    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779197    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779200    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779207    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779210    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779212    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779215    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779218    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:47:25.787595 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779221    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779223    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779226    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779229    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779231    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779234    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779237    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779240    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779242    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779245    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779248    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779250    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779253    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779256    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779258    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779261    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779264    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779266    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779270    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:47:25.788139 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779275    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779278    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779281    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779283    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779286    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779289    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779293    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779296    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779301    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779303    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779306    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779309    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779311    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779314    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779317    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779319    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779322    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779325    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779327    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:47:25.788592 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779330    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779333    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779336    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779339    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779341    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779344    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779347    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779349    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779352    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779354    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779357    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779359    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779362    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779364    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779367    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779370    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779372    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779375    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779377    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:47:25.789150 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779382    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779384    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779388    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779390    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779393    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779396    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779399    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779403    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779406    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779409    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779412    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779414    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779417    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779419    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779422    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779426    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779428    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779431    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779450    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:47:25.789604 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779454    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779457    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779460    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779463    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779466    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779469    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779472    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779474    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779477    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779479    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779482    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779484    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779487    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779492    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779495    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779499    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:47:25.790079 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.779526    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779561    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779565    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779568    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779571    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779574    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779577    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779579    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779582    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779585    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779588    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779590    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779593    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779596    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779599    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779602    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779605    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779608    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779610    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779613    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:47:25.790460 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779615    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779618    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779621    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779623    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779626    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779628    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779631    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779633    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779636    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779638    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779643    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779647    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779649    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779652    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779654    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779657    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779660    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779663    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779665    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779668    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:47:25.790943 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779670    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779673    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779676    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779678    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779681    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779685    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779689    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779693    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779696    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779698    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779701    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779703    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779706    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779709    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779712    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779714    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779717    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779719    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779722    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:47:25.791404 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779725    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779727    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779730    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779733    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779737    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779740    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779742    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779745    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779747    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779750    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779753    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779755    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779758    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779761    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779774    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779777    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779780    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779783    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779785    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.779788    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:47:25.791867 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780152    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780156    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780159    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780161    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780164    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780167    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780170    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780172    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780175    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780178    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780180    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780185    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780189    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780192    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780195    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780199    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780202    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780205    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780208    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:47:25.792355 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780211    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780214    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780217    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780220    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780222    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780225    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780229    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780231    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.780234    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.787798    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.787815    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.787832    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.787837    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787898    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787903    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787907    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787911    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787915    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787920    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:47:25.792829 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787923    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787926    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787929    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787932    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787934    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787937    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787940    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787943    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787951    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787954    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787957    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787960    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787964    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787966    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787969    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787972    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787975    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787978    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787980    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:47:25.793269 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787983    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787986    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787988    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787992    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787994    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.787997    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788000    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788002    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788005    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788007    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788010    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788013    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788015    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788018    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788027    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788031    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788034    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788036    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788039    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788042    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:47:25.793726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788044    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788048    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788051    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788054    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788057    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788059    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788062    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788065    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788068    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788070    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788073    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788076    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788079    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788081    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788084    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788086    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788089    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788092    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788094    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788097    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:47:25.794259 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788100    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788102    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788105    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788108    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788110    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788113    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788116    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788119    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788122    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788125    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788128    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788130    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788133    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788135    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788139    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788142    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788144    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788147    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788150    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:47:25.794726 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788153    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788155    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788158    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788161    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788164    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788166    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788169    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788171    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788174    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788176    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788179    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788181    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788184    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788187    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788189    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788192    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788195    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788197    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788201    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:47:25.795192 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788203    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788206    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788210    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.788239    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788377    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788382    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788385    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788388    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788391    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788394    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788397    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788400    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788403    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788405    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788408    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:47:25.795636 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788411    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788413    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788416    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788419    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788421    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788424    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788427    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788429    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788432    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788435    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788437    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788440    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788443    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788445    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788448    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788450    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788453    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788456    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788459    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788462    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:47:25.796011 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788464    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788467    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788470    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788472    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788475    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788477    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788480    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788484    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788488    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788490    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788493    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788495    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788498    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788501    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788503    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788506    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788509    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788511    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788514    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788517    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:47:25.796490 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788519    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788522    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788524    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788527    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788530    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788532    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788535    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788537    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788540    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788543    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788545    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788548    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788550    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788553    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788556    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788558    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788561    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788563    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788566    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788569    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:47:25.796971 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788573    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788575    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788578    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788581    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788584    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788586    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788590    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788593    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788596    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788599    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788602    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788605    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788607    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788610    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788613    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788616    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788618    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788621    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788624    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:47:25.797448 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788626    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788629    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788632    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788635    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788638    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788642    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788645    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788648    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788650    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788653    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788656    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788659    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788662    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788664    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788667    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788670    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:47:25.797909 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.788692    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788723    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788726    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788729    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788732    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788735    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788738    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788741    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788743    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788746    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788749    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788751    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788754    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788756    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788759    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788762    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788777    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788780    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788783    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788786    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:47:25.798289 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788789    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788791    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788794    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788797    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788800    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788802    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788805    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788808    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788811    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788814    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788816    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788820    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788822    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788825    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788828    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788830    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788833    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788836    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788838    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788841    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:47:25.798750 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788843    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788846    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788849    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788851    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788854    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788857    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788861    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788864    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788868    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788870    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788873    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788876    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788878    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788881    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788884    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788886    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788889    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788892    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788894    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788897    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:47:25.799248 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788900    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788902    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788905    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788908    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788911    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788915    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788918    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788922    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788925    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788927    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788930    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788932    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788935    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788938    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788941    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788943    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788946    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788949    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788953    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:47:25.799720 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788956    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788959    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788962    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788966    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788970    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788974    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788977    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788979    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788982    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788987    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788990    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788993    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788995    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.788998    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789001    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789003    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789006    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789009    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789013    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:47:25.800189 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789016    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789019    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789021    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789024    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789026    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789029    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789032    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789034    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:25.789037    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.789096    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.791875    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.792707    2578 server.go:1014] "Starting client certificate rotation"
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.792795    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.792827    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.794543    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 19:47:25.800637 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.797295    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 19:47:25.806140 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.806124    2578 log.go:25] "Validated CRI v1 runtime API"
Nov 02 19:47:25.809054 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.809038    2578 log.go:25] "Validated CRI v1 image API"
Nov 02 19:47:25.809866 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.809843    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 19:47:25.812557 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.812519    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 81737c7d-ecc7-4707-ba73-181ad48d46b2:/dev/nvme0n1p4 c20f6be1-c5e1-488e-a545-4754a5980b49:/dev/nvme0n1p3]
Nov 02 19:47:25.812645 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.812558    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 19:47:25.819898 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.819782    2578 manager.go:217] Machine: {Timestamp:2026-11-02 19:47:25.817810566 +0000 UTC m=+0.318869776 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100391 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2585e84e13d597d09a69b45e10b739 SystemUUID:ec2585e8-4e13-d597-d09a-69b45e10b739 BootID:4568f2d9-d301-40c7-a35a-ea1d6b8b206b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e3:e9:e7:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e3:e9:e7:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:7c:20:9e:45:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 19:47:25.820385 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.820374    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 19:47:25.820472 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.820460    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 19:47:25.824137 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.824107    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 19:47:25.824273 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.824139    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-196.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 19:47:25.824321 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.824276    2578 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 19:47:25.824321 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.824283    2578 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 19:47:25.824321 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.824313    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 19:47:25.824396 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.824331    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 19:47:25.825041 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.825031    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 19:47:25.828290 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.828278    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 19:47:25.828415 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.828406    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 19:47:25.829190 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.829179    2578 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 19:47:25.829220 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.829195    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 19:47:25.829220 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.829206    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 19:47:25.829220 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.829215    2578 kubelet.go:405] "Adding apiserver pod source"
Nov 02 19:47:25.829306 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.829223    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 19:47:25.829560 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.829547    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 19:47:25.831711 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.831688    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 19:47:25.837161 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.837146    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 19:47:25.838422 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.838410    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 19:47:25.838460 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.838440    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 19:47:25.840074 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.840062    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 19:47:25.840115 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.840080    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 19:47:25.840115 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.840086    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 19:47:25.840115 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.840091    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 19:47:25.840115 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.840098    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 19:47:25.840115 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.840108    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 19:47:25.840603 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.840594    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 19:47:25.840630 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.840604    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 19:47:25.840630 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.840612    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 19:47:25.840630 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.840617    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 19:47:25.840630 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.840626    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 19:47:25.840824 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.840635    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 19:47:25.841245 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.841234    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 19:47:25.841273 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.841253    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 19:47:25.845011 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.844998    2578 server.go:1263] "Started kubelet"
Nov 02 19:47:25.845231 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.845086    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 19:47:25.845292 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.845249    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 19:47:25.845343 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.845221    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 19:47:25.846024 ip-10-0-134-196 systemd[1]: Started Kubernetes Kubelet.
Nov 02 19:47:25.846429 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.846412    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 19:47:25.847735 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.847723    2578 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 19:47:25.850469 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:25.850429    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 19:47:25.854489 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.854470    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 19:47:25.854916 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.854901    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 19:47:25.855025 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:25.854985    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-196.us-west-2.compute.internal\" not found"
Nov 02 19:47:25.855586 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.855544    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 19:47:25.855586 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.855563    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 19:47:25.855701 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.855548    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 19:47:25.855737 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.855710    2578 factory.go:55] Registering systemd factory
Nov 02 19:47:25.855737 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.855727    2578 factory.go:223] Registration of the systemd container factory successfully
Nov 02 19:47:25.855833 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.855726    2578 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 19:47:25.855833 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.855788    2578 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 19:47:25.855974 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.855962    2578 factory.go:153] Registering CRI-O factory
Nov 02 19:47:25.856036 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.855978    2578 factory.go:223] Registration of the crio container factory successfully
Nov 02 19:47:25.856066 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.856028    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 19:47:25.856066 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.856050    2578 factory.go:103] Registering Raw factory
Nov 02 19:47:25.856066 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.856060    2578 manager.go:1199] Started watching for new ooms in manager
Nov 02 19:47:25.857249 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.857217    2578 manager.go:319] Starting recovery of all containers
Nov 02 19:47:25.869160 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.869136    2578 manager.go:324] Recovery completed
Nov 02 19:47:25.873635 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.873617    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:47:25.875567 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.875548    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:47:25.875647 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.875579    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:47:25.875647 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.875590    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:47:25.876545 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.876529    2578 cpu_manager.go:226] "Starting" policy="none"
Nov 02 19:47:25.876611 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.876544    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 19:47:25.876611 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.876564    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 19:47:25.876611 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.876537    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:47:25.877346 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.877330    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:47:25.877411 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.877368    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:47:25.877411 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.877378    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:47:25.878956 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.878944    2578 policy_none.go:50] "Start"
Nov 02 19:47:25.879007 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.878961    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 19:47:25.879007 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.878974    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 19:47:25.880718 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.880704    2578 policy_none.go:44] "Start"
Nov 02 19:47:25.905475 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.905444    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 19:47:25.907846 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.907831    2578 manager.go:341] "Starting Device Plugin manager"
Nov 02 19:47:25.907923 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:25.907875    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 19:47:25.907923 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.907885    2578 server.go:87] "Starting device plugin registration server"
Nov 02 19:47:25.908165 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.908151    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 19:47:25.908211 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.908182    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 19:47:25.908272 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.908257    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 19:47:25.909009 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.908350    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 19:47:25.909009 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.908375    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 19:47:25.909131 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:25.909046    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 19:47:25.909131 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:25.909095    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-196.us-west-2.compute.internal\" not found"
Nov 02 19:47:25.991670 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:25.991495    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-196.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 19:47:25.991670 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:25.991503    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-196.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 19:47:26.023699 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:25.991518    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-196.us-west-2.compute.internal.18e4521f4460fc90  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-196.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-196.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:47:25.84497064 +0000 UTC m=+0.346029849,LastTimestamp:2026-11-02 19:47:25.84497064 +0000 UTC m=+0.346029849,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-196.us-west-2.compute.internal,}"
Nov 02 19:47:26.023699 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.009043    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:47:26.023699 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.009914    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:47:26.023699 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.009939    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:47:26.023699 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.009958    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:47:26.023699 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.009990    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.053732 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.053697    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 19:47:26.054901 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.054881    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 19:47:26.054901 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.054901    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 19:47:26.055081 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.054918    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 19:47:26.055081 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:26.054958    2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 19:47:26.074713 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.074685    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 19:47:26.074823 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.074751    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n2bl8"
Nov 02 19:47:26.138193 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.138164    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:47:26.140092 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.140075    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:47:26.155169 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.155134    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"]
Nov 02 19:47:26.155226 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.155206    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:47:26.158030 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.158015    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:47:26.158105 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.158046    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:47:26.158105 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.158062    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:47:26.160364 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.160352    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:47:26.160531 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.160517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.160568 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.160548    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:47:26.161156 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.161142    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:47:26.161221 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.161173    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:47:26.161221 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.161183    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:47:26.161288 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.161144    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:47:26.161288 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.161244    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:47:26.161288 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.161257    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:47:26.163349 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.163331    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.163449 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.163361    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:47:26.164090 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.164072    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:47:26.164193 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.164099    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:47:26.164193 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.164110    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:47:26.188283 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:26.188266    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-196.us-west-2.compute.internal\" not found" node="ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.191369 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:26.191353    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-196.us-west-2.compute.internal\" not found" node="ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.231409 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.231388    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:47:26.232366 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.232348    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.232439 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:26.232380    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-196.us-west-2.compute.internal\": node \"ip-10-0-134-196.us-west-2.compute.internal\" not found"
Nov 02 19:47:26.233133 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.233115    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:47:26.256085 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.256063    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.257191 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.257172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f28b16d2fb7687a2bf2bca5db4901e34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"f28b16d2fb7687a2bf2bca5db4901e34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.257253 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.257198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f28b16d2fb7687a2bf2bca5db4901e34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"f28b16d2fb7687a2bf2bca5db4901e34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.257253 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.257216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09b897c9011dced250aef3fd51f6bd3f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"09b897c9011dced250aef3fd51f6bd3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.312521 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.312445    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:47:26.313394 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.313380    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.357880 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.357847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f28b16d2fb7687a2bf2bca5db4901e34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"f28b16d2fb7687a2bf2bca5db4901e34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.357880 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.357881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09b897c9011dced250aef3fd51f6bd3f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"09b897c9011dced250aef3fd51f6bd3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.358069 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.357904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f28b16d2fb7687a2bf2bca5db4901e34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"f28b16d2fb7687a2bf2bca5db4901e34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.358069 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.357960    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f28b16d2fb7687a2bf2bca5db4901e34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"f28b16d2fb7687a2bf2bca5db4901e34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.358069 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.357960    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f28b16d2fb7687a2bf2bca5db4901e34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"f28b16d2fb7687a2bf2bca5db4901e34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.358069 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.357968    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09b897c9011dced250aef3fd51f6bd3f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"09b897c9011dced250aef3fd51f6bd3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.366757 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.366736    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:47:26.491954 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.491921    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.494067 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.494050    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:47:26.795078 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.795034    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 19:47:26.795681 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.795308    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 19:47:26.795681 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.795322    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 19:47:26.795681 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.795347    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 19:47:26.795681 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:26.795346    2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a494c12af1f134ca494febab804ea946-87fdf5767f91ce37.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-196.us-west-2.compute.internal.18e4521f4633d42c\": read tcp 10.0.134.196:58208->3.132.191.132:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-196.us-west-2.compute.internal.18e4521f4633d42c  default   5349 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-196.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-196.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-196.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:47:25 +0000 UTC,LastTimestamp:2026-11-02 19:47:26.158029309 +0000 UTC m=+0.659088518,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-196.us-west-2.compute.internal,}"
Nov 02 19:47:26.830320 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.830292    2578 apiserver.go:52] "Watching apiserver"
Nov 02 19:47:26.854699 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:26.854674    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 19:47:27.080697 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.080602    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:42:26 +0000 UTC" deadline="2028-06-11 06:16:33.71102866 +0000 UTC"
Nov 02 19:47:27.080697 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.080646    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14074h29m6.630386966s"
Nov 02 19:47:27.131083 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.131051    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 19:47:27.131401 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.131382    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-jwvp7","kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal","openshift-image-registry/node-ca-qxbct","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal","openshift-multus/multus-rsl26","kube-system/global-pull-secret-syncer-kf87v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k","openshift-network-diagnostics/network-check-target-hxstf","openshift-network-operator/iptables-alerter-k46lq","openshift-dns/node-resolver-h6nfx","openshift-multus/network-metrics-daemon-qcr2p","openshift-ovn-kubernetes/ovnkube-node-w4m6t","openshift-cluster-node-tuning-operator/tuned-mm2bm","openshift-multus/multus-additional-cni-plugins-h6b6p"]
Nov 02 19:47:27.134051 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.134032    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:47:27.137788 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.137753    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:47:27.139914 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.139900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.142101 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.142086    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:27.142160 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.142143    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:27.144264 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.144249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.146454 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.146436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:27.146532 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.146495    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:27.148681 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.148665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:47:27.151682 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.151669    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:47:27.154012 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.153998    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:27.154060 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.154047    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:27.156521 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.156500    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.158812 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.158791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.161125 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.161107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.163458 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.163558 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89wdg\" (UniqueName: \"kubernetes.io/projected/857aedf0-97a2-469a-b0fd-765845566e58-kube-api-access-89wdg\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.163558 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af20298-29d1-41e2-8714-dbd4367aa89c-iptables-alerter-script\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:47:27.163558 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-kubelet\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.163558 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-var-lib-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.163766 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-node-log\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.163766 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmxqn\" (UniqueName: \"kubernetes.io/projected/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-kube-api-access-gmxqn\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:47:27.163766 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-cni-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.163766 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bd5801e-0707-4269-aef3-c22008c52025-cni-binary-copy\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.163766 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd0732dd-9b73-4204-a64b-ba95c2aa6007-dbus\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:27.163766 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqgql\" (UniqueName: \"kubernetes.io/projected/3bd5801e-0707-4269-aef3-c22008c52025-kube-api-access-pqgql\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.164092 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.164092 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knqwv\" (UniqueName: \"kubernetes.io/projected/9c982442-0aef-45b4-acd0-16e707422ca9-kube-api-access-knqwv\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.164092 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/086392f7-3abd-4c9b-ae7b-69ccd1985d1b-konnectivity-ca\") pod \"konnectivity-agent-jwvp7\" (UID: \"086392f7-3abd-4c9b-ae7b-69ccd1985d1b\") " pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:47:27.164092 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-ovnkube-config\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.164092 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af20298-29d1-41e2-8714-dbd4367aa89c-host-slash\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:47:27.164092 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cmbv\" (UniqueName: \"kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv\") pod \"network-check-target-hxstf\" (UID: \"256ea3f6-c170-4734-a39d-63fcc9049b83\") " pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:27.164092 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-etc-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.164092 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.163997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-ovn\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.164092 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-systemd\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.164092 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.164092 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:27.164092 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-registration-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-slash\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-ovnkube-script-lib\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-cni-netd\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-host\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-serviceca\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-hostroot\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-etc-kubernetes\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls4kl\" (UniqueName: \"kubernetes.io/projected/539f14e3-83e4-4288-af2e-6dbd929a1565-kube-api-access-ls4kl\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-cni-bin\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-env-overrides\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-k8s-cni-cncf-io\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-netns\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.164610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-tmp-dir\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c982442-0aef-45b4-acd0-16e707422ca9-ovn-node-metrics-cert\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164624    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/086392f7-3abd-4c9b-ae7b-69ccd1985d1b-agent-certs\") pod \"konnectivity-agent-jwvp7\" (UID: \"086392f7-3abd-4c9b-ae7b-69ccd1985d1b\") " pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-cnibin\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsss9\" (UniqueName: \"kubernetes.io/projected/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-kube-api-access-zsss9\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-socket-dir-parent\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-cni-multus\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-kubelet\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd0732dd-9b73-4204-a64b-ba95c2aa6007-kubelet-config\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pj2b\" (UniqueName: \"kubernetes.io/projected/3af20298-29d1-41e2-8714-dbd4367aa89c-kube-api-access-4pj2b\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-socket-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-device-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-hosts-file\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-run-netns\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-cni-bin\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.164976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-conf-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.165257 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.165006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-multus-certs\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.165709 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.165040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-systemd-units\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.165709 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.165061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-log-socket\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.165709 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.165087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-system-cni-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.165709 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.165103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-sys-fs\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.165709 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.165126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3bd5801e-0707-4269-aef3-c22008c52025-multus-daemon-config\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.191000 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.190972    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 19:47:27.191000 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.190985    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 19:47:27.191000 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.190981    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7zhgs\""
Nov 02 19:47:27.194082 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.194068    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 19:47:27.194130 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.194096    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 19:47:27.194166 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.194133    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p97wz\""
Nov 02 19:47:27.194282 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.194270    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 19:47:27.196371 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.196349    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 19:47:27.257415 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.257392    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 19:47:27.265798 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.265768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmxqn\" (UniqueName: \"kubernetes.io/projected/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-kube-api-access-gmxqn\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:47:27.265912 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.265806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-cni-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.265912 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.265822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bd5801e-0707-4269-aef3-c22008c52025-cni-binary-copy\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.265912 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.265838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd0732dd-9b73-4204-a64b-ba95c2aa6007-dbus\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:27.265912 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.265864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysctl-d\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.265912 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.265880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-systemd\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.265912 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.265897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqgql\" (UniqueName: \"kubernetes.io/projected/3bd5801e-0707-4269-aef3-c22008c52025-kube-api-access-pqgql\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.265912 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.265914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.266226 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.265980    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.266226 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266000    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-cni-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.266226 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266008    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd0732dd-9b73-4204-a64b-ba95c2aa6007-dbus\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:27.266226 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqwv\" (UniqueName: \"kubernetes.io/projected/9c982442-0aef-45b4-acd0-16e707422ca9-kube-api-access-knqwv\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.266226 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/086392f7-3abd-4c9b-ae7b-69ccd1985d1b-konnectivity-ca\") pod \"konnectivity-agent-jwvp7\" (UID: \"086392f7-3abd-4c9b-ae7b-69ccd1985d1b\") " pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:47:27.266226 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-ovnkube-config\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.266481 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-lib-modules\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.266481 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af20298-29d1-41e2-8714-dbd4367aa89c-host-slash\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:47:27.266481 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266369    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af20298-29d1-41e2-8714-dbd4367aa89c-host-slash\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:47:27.266481 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmbv\" (UniqueName: \"kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv\") pod \"network-check-target-hxstf\" (UID: \"256ea3f6-c170-4734-a39d-63fcc9049b83\") " pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:27.266481 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-etc-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.266481 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-ovn\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.266481 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-modprobe-d\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266482    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-etc-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266513    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-ovn\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-systemd\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266608    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-systemd\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266625    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266652    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.266687    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266697    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/086392f7-3abd-4c9b-ae7b-69ccd1985d1b-konnectivity-ca\") pod \"konnectivity-agent-jwvp7\" (UID: \"086392f7-3abd-4c9b-ae7b-69ccd1985d1b\") " pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cnibin\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.266793 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.266769    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs podName:539f14e3-83e4-4288-af2e-6dbd929a1565 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:27.766723981 +0000 UTC m=+2.267783199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs") pod "network-metrics-daemon-qcr2p" (UID: "539f14e3-83e4-4288-af2e-6dbd929a1565") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-registration-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-slash\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-ovnkube-script-lib\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-system-cni-dir\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266941    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-registration-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266947    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-slash\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-cni-netd\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-host\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266984    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-cni-netd\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.267002    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-host\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.267007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysctl-conf\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.267031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-serviceca\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.267055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-hostroot\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.267072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-etc-kubernetes\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.267087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.267104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls4kl\" (UniqueName: \"kubernetes.io/projected/539f14e3-83e4-4288-af2e-6dbd929a1565-kube-api-access-ls4kl\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:27.267342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.267112    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-hostroot\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.267988 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.267129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-cni-bin\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.267988 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.267130    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-etc-kubernetes\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.267988 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.267183    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-cni-bin\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.267988 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.266724    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.267988 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.267283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-env-overrides\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.267988 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.267927    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:27.268250 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.268005    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret podName:cd0732dd-9b73-4204-a64b-ba95c2aa6007 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:27.767986416 +0000 UTC m=+2.269045624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret") pod "global-pull-secret-syncer-kf87v" (UID: "cd0732dd-9b73-4204-a64b-ba95c2aa6007") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:27.268250 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.268042    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-serviceca\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:47:27.269922 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.269893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9201398f-e237-4997-9091-ffe08b8291ad-tmp\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.270020 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.269987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-k8s-cni-cncf-io\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.270084 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270027    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-k8s-cni-cncf-io\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.270084 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-netns\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.270177 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-tmp-dir\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:47:27.270226 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c982442-0aef-45b4-acd0-16e707422ca9-ovn-node-metrics-cert\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.270528 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270505    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/086392f7-3abd-4c9b-ae7b-69ccd1985d1b-agent-certs\") pod \"konnectivity-agent-jwvp7\" (UID: \"086392f7-3abd-4c9b-ae7b-69ccd1985d1b\") " pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:47:27.270604 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysconfig\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.270643 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-kubernetes\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.270677 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270656    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-netns\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.270677 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.270752 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-cnibin\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.270799 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsss9\" (UniqueName: \"kubernetes.io/projected/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-kube-api-access-zsss9\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:47:27.270799 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270786    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-cnibin\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.270916 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9201398f-e237-4997-9091-ffe08b8291ad-etc-tuned\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.270977 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.270958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cni-binary-copy\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.271346 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.271400 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62972\" (UniqueName: \"kubernetes.io/projected/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-kube-api-access-62972\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.271455 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-socket-dir-parent\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.271455 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271401    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-tmp-dir\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:47:27.271455 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-cni-multus\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.271600 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-kubelet\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.271600 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271497    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-kubelet\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.271600 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271505    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd0732dd-9b73-4204-a64b-ba95c2aa6007-kubelet-config\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:27.271600 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pj2b\" (UniqueName: \"kubernetes.io/projected/3af20298-29d1-41e2-8714-dbd4367aa89c-kube-api-access-4pj2b\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:47:27.271600 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-var-lib-kubelet\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.271600 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271583    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 19:47:27.271600 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271594    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-cni-multus\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.271930 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-socket-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.271930 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271662    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-socket-dir-parent\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.271930 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271674    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd0732dd-9b73-4204-a64b-ba95c2aa6007-kubelet-config\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:27.271930 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-device-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.271930 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-hosts-file\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:47:27.271930 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271780    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-socket-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.271930 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-run-netns\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.271930 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271846    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-device-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.271930 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-run\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.271930 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-cni-bin\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.271930 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271907    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-hosts-file\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:47:27.271930 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-conf-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271938    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-run-netns\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-multus-certs\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271988    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-conf-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-systemd-units\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.271968    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-cni-bin\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-log-socket\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272014    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-multus-certs\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-host\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272071    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-systemd-units\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-system-cni-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272112    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-log-socket\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-sys-fs\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-sys\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272203    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-system-cni-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272221    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-sys-fs\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3bd5801e-0707-4269-aef3-c22008c52025-multus-daemon-config\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.272440 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.273130 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89wdg\" (UniqueName: \"kubernetes.io/projected/857aedf0-97a2-469a-b0fd-765845566e58-kube-api-access-89wdg\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.273130 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af20298-29d1-41e2-8714-dbd4367aa89c-iptables-alerter-script\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:47:27.273130 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-kubelet\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.273130 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-var-lib-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.273130 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-node-log\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.273130 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272450    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.273130 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhd5m\" (UniqueName: \"kubernetes.io/projected/9201398f-e237-4997-9091-ffe08b8291ad-kube-api-access-dhd5m\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.273130 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272553    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-node-log\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.273130 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272584    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-kubelet\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.273130 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-var-lib-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.273130 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.272706    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3bd5801e-0707-4269-aef3-c22008c52025-multus-daemon-config\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.275466 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.275443    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/086392f7-3abd-4c9b-ae7b-69ccd1985d1b-agent-certs\") pod \"konnectivity-agent-jwvp7\" (UID: \"086392f7-3abd-4c9b-ae7b-69ccd1985d1b\") " pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:47:27.296361 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.296334    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 19:47:27.296450 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.296379    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9bj9s"
Nov 02 19:47:27.300896 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.300877    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 19:47:27.301046 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301031    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:47:27.301106 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301036    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 19:47:27.301256 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301239    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 19:47:27.301335 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301306    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 19:47:27.301394 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301346    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 19:47:27.301394 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301359    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 19:47:27.301394 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301377    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fhf7l\""
Nov 02 19:47:27.301394 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301389    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:47:27.301596 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301426    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 19:47:27.301596 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301451    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6t9zq\""
Nov 02 19:47:27.301596 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301450    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 19:47:27.301596 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301483    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 19:47:27.301596 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301498    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 19:47:27.301596 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301500    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 19:47:27.301596 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301474    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 19:47:27.301909 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301638    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 19:47:27.301909 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301664    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 19:47:27.301909 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301674    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 19:47:27.301909 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301681    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 19:47:27.301909 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301723    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7q2qk\""
Nov 02 19:47:27.301909 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301748    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 19:47:27.301909 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.301793    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 19:47:27.303207 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.303181    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af20298-29d1-41e2-8714-dbd4367aa89c-iptables-alerter-script\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:47:27.307831 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.307813    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bd5801e-0707-4269-aef3-c22008c52025-cni-binary-copy\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.307916 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.307873    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-ovnkube-config\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.308545 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.308524    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-env-overrides\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.309388 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.309372    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xk7lc\""
Nov 02 19:47:27.309388 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.309374    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x9wbn\""
Nov 02 19:47:27.309570 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.309552    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qzg2s\""
Nov 02 19:47:27.309663 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.309592    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 19:47:27.309663 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.309596    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rmffj\""
Nov 02 19:47:27.312684 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.312661    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c982442-0aef-45b4-acd0-16e707422ca9-ovn-node-metrics-cert\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.318220 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.318202    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-ovnkube-script-lib\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.330562 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.330535    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqgql\" (UniqueName: \"kubernetes.io/projected/3bd5801e-0707-4269-aef3-c22008c52025-kube-api-access-pqgql\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.330740 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.330706    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmxqn\" (UniqueName: \"kubernetes.io/projected/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-kube-api-access-gmxqn\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:47:27.350926 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.350898    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:47:27.350926 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.350924    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:47:27.351077 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.350938    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cmbv for pod openshift-network-diagnostics/network-check-target-hxstf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:27.351077 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.350999    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv podName:256ea3f6-c170-4734-a39d-63fcc9049b83 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:27.850979757 +0000 UTC m=+2.352038981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmbv" (UniqueName: "kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv") pod "network-check-target-hxstf" (UID: "256ea3f6-c170-4734-a39d-63fcc9049b83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:27.353091 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.353066    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsss9\" (UniqueName: \"kubernetes.io/projected/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-kube-api-access-zsss9\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:47:27.353184 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.353090    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqwv\" (UniqueName: \"kubernetes.io/projected/9c982442-0aef-45b4-acd0-16e707422ca9-kube-api-access-knqwv\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.353184 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.353099    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89wdg\" (UniqueName: \"kubernetes.io/projected/857aedf0-97a2-469a-b0fd-765845566e58-kube-api-access-89wdg\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.353311 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.353298    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pj2b\" (UniqueName: \"kubernetes.io/projected/3af20298-29d1-41e2-8714-dbd4367aa89c-kube-api-access-4pj2b\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:47:27.357056 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.357036    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls4kl\" (UniqueName: \"kubernetes.io/projected/539f14e3-83e4-4288-af2e-6dbd929a1565-kube-api-access-ls4kl\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:27.373085 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysconfig\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.373171 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-kubernetes\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.373171 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.373171 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9201398f-e237-4997-9091-ffe08b8291ad-etc-tuned\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.373171 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cni-binary-copy\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.373363 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373184    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-kubernetes\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.373363 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373190    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysconfig\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.373363 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.373363 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62972\" (UniqueName: \"kubernetes.io/projected/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-kube-api-access-62972\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.373574 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-var-lib-kubelet\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.373574 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-run\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.373574 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-host\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.373574 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-sys\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.373574 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373461    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-var-lib-kubelet\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.373574 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.373574 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhd5m\" (UniqueName: \"kubernetes.io/projected/9201398f-e237-4997-9091-ffe08b8291ad-kube-api-access-dhd5m\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.373574 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373530    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-host\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.373574 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysctl-d\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.374038 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-systemd\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.374038 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373585    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-run\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.374038 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-lib-modules\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.374038 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-modprobe-d\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.374038 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373699    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-sys\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.374038 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373724    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-systemd\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.374038 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cnibin\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.374038 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-system-cni-dir\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.374038 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysctl-conf\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.374038 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9201398f-e237-4997-9091-ffe08b8291ad-tmp\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.374038 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.373954    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-modprobe-d\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.374038 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.374028    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-system-cni-dir\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.374472 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.374075    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cnibin\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.374472 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.374078    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysctl-d\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.374472 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.374138    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-lib-modules\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.374472 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.374205    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysctl-conf\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.374472 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.374310    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.374472 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.374318    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.374472 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.374442    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cni-binary-copy\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.374714 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.374594    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.375366 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.375348    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9201398f-e237-4997-9091-ffe08b8291ad-etc-tuned\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.375984 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.375965    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9201398f-e237-4997-9091-ffe08b8291ad-tmp\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.435785 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.435749    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhd5m\" (UniqueName: \"kubernetes.io/projected/9201398f-e237-4997-9091-ffe08b8291ad-kube-api-access-dhd5m\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.436250 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.436231    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62972\" (UniqueName: \"kubernetes.io/projected/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-kube-api-access-62972\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.471035 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.471002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:47:27.486550 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.486516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:47:27.500825 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.500791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rsl26"
Nov 02 19:47:27.517071 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.517040    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:47:27.517218 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.517113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:47:27.540905 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.540874    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:47:27.568165 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.568131    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:47:27.576580 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.576551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:27.591820 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.591791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:47:27.595621 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:27.595590    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b897c9011dced250aef3fd51f6bd3f.slice/crio-f43d752c2ae6f2bf739bf9ce3c576f5bfdf1445401edd1a5576a254289b579e4 WatchSource:0}: Error finding container f43d752c2ae6f2bf739bf9ce3c576f5bfdf1445401edd1a5576a254289b579e4: Status 404 returned error can't find the container with id f43d752c2ae6f2bf739bf9ce3c576f5bfdf1445401edd1a5576a254289b579e4
Nov 02 19:47:27.596148 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:27.596125    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9201398f_e237_4997_9091_ffe08b8291ad.slice/crio-00f7cb0ec8016f7928bd203aa976a6a6aef3752d7243b691462c5fcade3d3c24 WatchSource:0}: Error finding container 00f7cb0ec8016f7928bd203aa976a6a6aef3752d7243b691462c5fcade3d3c24: Status 404 returned error can't find the container with id 00f7cb0ec8016f7928bd203aa976a6a6aef3752d7243b691462c5fcade3d3c24
Nov 02 19:47:27.597055 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:27.597024    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc7f8e6_1a74_463a_9d86_2fc565f3fe0f.slice/crio-bf2c35c34c369e2adfb593d43f1de8f796386378a20f11e0170c91e72da661db WatchSource:0}: Error finding container bf2c35c34c369e2adfb593d43f1de8f796386378a20f11e0170c91e72da661db: Status 404 returned error can't find the container with id bf2c35c34c369e2adfb593d43f1de8f796386378a20f11e0170c91e72da661db
Nov 02 19:47:27.603363 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.603334    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:47:27.604534 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:27.604508    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c982442_0aef_45b4_acd0_16e707422ca9.slice/crio-cef2a9f7284bde86be5b82b9bf7f686b02392a0bd023f4e1cff15bb80a174a65 WatchSource:0}: Error finding container cef2a9f7284bde86be5b82b9bf7f686b02392a0bd023f4e1cff15bb80a174a65: Status 404 returned error can't find the container with id cef2a9f7284bde86be5b82b9bf7f686b02392a0bd023f4e1cff15bb80a174a65
Nov 02 19:47:27.607400 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:27.607358    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce51ba2f_b8b5_4b53_8a8e_be70bda4df63.slice/crio-376af49dc92e8b997d83a2035414e5bc40a614310b1bf676d2af0767e7592e92 WatchSource:0}: Error finding container 376af49dc92e8b997d83a2035414e5bc40a614310b1bf676d2af0767e7592e92: Status 404 returned error can't find the container with id 376af49dc92e8b997d83a2035414e5bc40a614310b1bf676d2af0767e7592e92
Nov 02 19:47:27.608208 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:27.608184    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28b16d2fb7687a2bf2bca5db4901e34.slice/crio-6ca0a98c88ac99c89ccd69f236d073ecfeccf28369fa4e589b6887d032d308ae WatchSource:0}: Error finding container 6ca0a98c88ac99c89ccd69f236d073ecfeccf28369fa4e589b6887d032d308ae: Status 404 returned error can't find the container with id 6ca0a98c88ac99c89ccd69f236d073ecfeccf28369fa4e589b6887d032d308ae
Nov 02 19:47:27.720752 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.720721    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:47:27.777008 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.776978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:27.777198 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.777020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:27.777198 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.777126    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:27.777198 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.777184    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret podName:cd0732dd-9b73-4204-a64b-ba95c2aa6007 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:28.777167255 +0000 UTC m=+3.278226468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret") pod "global-pull-secret-syncer-kf87v" (UID: "cd0732dd-9b73-4204-a64b-ba95c2aa6007") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:27.777198 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.777126    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:27.777375 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.777237    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs podName:539f14e3-83e4-4288-af2e-6dbd929a1565 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:28.777226791 +0000 UTC m=+3.278286005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs") pod "network-metrics-daemon-qcr2p" (UID: "539f14e3-83e4-4288-af2e-6dbd929a1565") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:27.838679 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.838601    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:47:27.877997 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:27.877964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmbv\" (UniqueName: \"kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv\") pod \"network-check-target-hxstf\" (UID: \"256ea3f6-c170-4734-a39d-63fcc9049b83\") " pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:27.878144 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.878127    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:47:27.878187 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.878148    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:47:27.878187 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.878158    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cmbv for pod openshift-network-diagnostics/network-check-target-hxstf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:27.878258 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:27.878211    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv podName:256ea3f6-c170-4734-a39d-63fcc9049b83 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:28.878193796 +0000 UTC m=+3.379253011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmbv" (UniqueName: "kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv") pod "network-check-target-hxstf" (UID: "256ea3f6-c170-4734-a39d-63fcc9049b83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:28.058964 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:28.058914    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal" event={"ID":"f28b16d2fb7687a2bf2bca5db4901e34","Type":"ContainerStarted","Data":"6ca0a98c88ac99c89ccd69f236d073ecfeccf28369fa4e589b6887d032d308ae"}
Nov 02 19:47:28.059992 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:28.059965    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6b6p" event={"ID":"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63","Type":"ContainerStarted","Data":"376af49dc92e8b997d83a2035414e5bc40a614310b1bf676d2af0767e7592e92"}
Nov 02 19:47:28.061061 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:28.061042    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t" event={"ID":"9c982442-0aef-45b4-acd0-16e707422ca9","Type":"ContainerStarted","Data":"cef2a9f7284bde86be5b82b9bf7f686b02392a0bd023f4e1cff15bb80a174a65"}
Nov 02 19:47:28.062066 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:28.062048    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mm2bm" event={"ID":"9201398f-e237-4997-9091-ffe08b8291ad","Type":"ContainerStarted","Data":"00f7cb0ec8016f7928bd203aa976a6a6aef3752d7243b691462c5fcade3d3c24"}
Nov 02 19:47:28.063001 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:28.062980    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal" event={"ID":"09b897c9011dced250aef3fd51f6bd3f","Type":"ContainerStarted","Data":"f43d752c2ae6f2bf739bf9ce3c576f5bfdf1445401edd1a5576a254289b579e4"}
Nov 02 19:47:28.063918 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:28.063899    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h6nfx" event={"ID":"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f","Type":"ContainerStarted","Data":"bf2c35c34c369e2adfb593d43f1de8f796386378a20f11e0170c91e72da661db"}
Nov 02 19:47:28.127094 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:28.127028    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:47:28.299779 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:28.299722    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:42:27 +0000 UTC" deadline="2028-05-25 16:06:19.679187277 +0000 UTC"
Nov 02 19:47:28.299948 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:28.299787    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13676h18m51.379416946s"
Nov 02 19:47:28.355035 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:28.355005    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220ffe56_58a3_4bd9_b93a_dcb1ac38fc89.slice/crio-1e45fb69a621bbcdb227e674c845715578ebcfb32dae085df43b92e7cb232f81 WatchSource:0}: Error finding container 1e45fb69a621bbcdb227e674c845715578ebcfb32dae085df43b92e7cb232f81: Status 404 returned error can't find the container with id 1e45fb69a621bbcdb227e674c845715578ebcfb32dae085df43b92e7cb232f81
Nov 02 19:47:28.366956 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:28.366928    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086392f7_3abd_4c9b_ae7b_69ccd1985d1b.slice/crio-5cd4129444488cb4bbda1fe3de110c895aabfef870d9ecf7e623276b6254b9d5 WatchSource:0}: Error finding container 5cd4129444488cb4bbda1fe3de110c895aabfef870d9ecf7e623276b6254b9d5: Status 404 returned error can't find the container with id 5cd4129444488cb4bbda1fe3de110c895aabfef870d9ecf7e623276b6254b9d5
Nov 02 19:47:28.375523 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:28.375496    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd5801e_0707_4269_aef3_c22008c52025.slice/crio-a3f556e8af1c4fbc9621ccb3f4727aeba79f2f37ee2915aee1d346972f2ddeee WatchSource:0}: Error finding container a3f556e8af1c4fbc9621ccb3f4727aeba79f2f37ee2915aee1d346972f2ddeee: Status 404 returned error can't find the container with id a3f556e8af1c4fbc9621ccb3f4727aeba79f2f37ee2915aee1d346972f2ddeee
Nov 02 19:47:28.412424 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:28.412375    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af20298_29d1_41e2_8714_dbd4367aa89c.slice/crio-ceb2463c2c1b55a51d6e729b7c3fae2f8b825354b8b0276972ba8b29a2a534dc WatchSource:0}: Error finding container ceb2463c2c1b55a51d6e729b7c3fae2f8b825354b8b0276972ba8b29a2a534dc: Status 404 returned error can't find the container with id ceb2463c2c1b55a51d6e729b7c3fae2f8b825354b8b0276972ba8b29a2a534dc
Nov 02 19:47:28.438841 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:47:28.438811    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857aedf0_97a2_469a_b0fd_765845566e58.slice/crio-0014cc2b5f0c8e635f6777129ed2635b92c38a513e307a6b15f8e5b4d05c41df WatchSource:0}: Error finding container 0014cc2b5f0c8e635f6777129ed2635b92c38a513e307a6b15f8e5b4d05c41df: Status 404 returned error can't find the container with id 0014cc2b5f0c8e635f6777129ed2635b92c38a513e307a6b15f8e5b4d05c41df
Nov 02 19:47:28.783152 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:28.783109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:28.783348 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:28.783201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:28.783429 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:28.783408    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:28.783496 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:28.783483    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret podName:cd0732dd-9b73-4204-a64b-ba95c2aa6007 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:30.783462086 +0000 UTC m=+5.284521300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret") pod "global-pull-secret-syncer-kf87v" (UID: "cd0732dd-9b73-4204-a64b-ba95c2aa6007") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:28.783581 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:28.783569    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:28.783642 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:28.783611    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs podName:539f14e3-83e4-4288-af2e-6dbd929a1565 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:30.783598202 +0000 UTC m=+5.284657416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs") pod "network-metrics-daemon-qcr2p" (UID: "539f14e3-83e4-4288-af2e-6dbd929a1565") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:28.883661 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:28.883620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmbv\" (UniqueName: \"kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv\") pod \"network-check-target-hxstf\" (UID: \"256ea3f6-c170-4734-a39d-63fcc9049b83\") " pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:28.884327 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:28.883799    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:47:28.884327 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:28.883834    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:47:28.884327 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:28.883848    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cmbv for pod openshift-network-diagnostics/network-check-target-hxstf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:28.884327 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:28.883908    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv podName:256ea3f6-c170-4734-a39d-63fcc9049b83 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:30.883890342 +0000 UTC m=+5.384949557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmbv" (UniqueName: "kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv") pod "network-check-target-hxstf" (UID: "256ea3f6-c170-4734-a39d-63fcc9049b83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:29.055688 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:29.055599    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:29.055905 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:29.055741    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:29.055905 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:29.055855    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:29.056033 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:29.055943    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:29.056033 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:29.056013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:29.056148 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:29.056096    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:29.073020 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:29.072988    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qxbct" event={"ID":"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89","Type":"ContainerStarted","Data":"1e45fb69a621bbcdb227e674c845715578ebcfb32dae085df43b92e7cb232f81"}
Nov 02 19:47:29.075402 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:29.075365    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k" event={"ID":"857aedf0-97a2-469a-b0fd-765845566e58","Type":"ContainerStarted","Data":"0014cc2b5f0c8e635f6777129ed2635b92c38a513e307a6b15f8e5b4d05c41df"}
Nov 02 19:47:29.078208 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:29.078174    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k46lq" event={"ID":"3af20298-29d1-41e2-8714-dbd4367aa89c","Type":"ContainerStarted","Data":"ceb2463c2c1b55a51d6e729b7c3fae2f8b825354b8b0276972ba8b29a2a534dc"}
Nov 02 19:47:29.081825 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:29.081796    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rsl26" event={"ID":"3bd5801e-0707-4269-aef3-c22008c52025","Type":"ContainerStarted","Data":"a3f556e8af1c4fbc9621ccb3f4727aeba79f2f37ee2915aee1d346972f2ddeee"}
Nov 02 19:47:29.084236 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:29.084212    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jwvp7" event={"ID":"086392f7-3abd-4c9b-ae7b-69ccd1985d1b","Type":"ContainerStarted","Data":"5cd4129444488cb4bbda1fe3de110c895aabfef870d9ecf7e623276b6254b9d5"}
Nov 02 19:47:29.300594 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:29.300551    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:42:27 +0000 UTC" deadline="2028-05-03 20:05:36.369000167 +0000 UTC"
Nov 02 19:47:29.300594 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:29.300594    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13152h18m7.068410744s"
Nov 02 19:47:30.798198 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:30.797692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:30.798198 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:30.797753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:30.798198 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:30.797953    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:30.798198 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:30.798020    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret podName:cd0732dd-9b73-4204-a64b-ba95c2aa6007 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:34.798000919 +0000 UTC m=+9.299060121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret") pod "global-pull-secret-syncer-kf87v" (UID: "cd0732dd-9b73-4204-a64b-ba95c2aa6007") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:30.798198 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:30.798090    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:30.798198 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:30.798125    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs podName:539f14e3-83e4-4288-af2e-6dbd929a1565 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:34.798113831 +0000 UTC m=+9.299173046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs") pod "network-metrics-daemon-qcr2p" (UID: "539f14e3-83e4-4288-af2e-6dbd929a1565") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:30.899090 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:30.898728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmbv\" (UniqueName: \"kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv\") pod \"network-check-target-hxstf\" (UID: \"256ea3f6-c170-4734-a39d-63fcc9049b83\") " pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:30.899090 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:30.898955    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:47:30.899090 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:30.898978    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:47:30.899090 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:30.898992    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cmbv for pod openshift-network-diagnostics/network-check-target-hxstf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:30.899090 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:30.899060    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv podName:256ea3f6-c170-4734-a39d-63fcc9049b83 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:34.899041923 +0000 UTC m=+9.400101126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmbv" (UniqueName: "kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv") pod "network-check-target-hxstf" (UID: "256ea3f6-c170-4734-a39d-63fcc9049b83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:31.056267 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:31.056229    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:31.056475 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:31.056377    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:31.056475 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:31.056465    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:31.056654 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:31.056540    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:31.056654 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:31.056601    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:31.056761 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:31.056702    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:32.116806 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:32.116560    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" containerID="987d8f6f5d272f60c9da42195a68fdf932bebe89289503a81606f41342785026" exitCode=0
Nov 02 19:47:32.116806 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:32.116699    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6b6p" event={"ID":"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63","Type":"ContainerDied","Data":"987d8f6f5d272f60c9da42195a68fdf932bebe89289503a81606f41342785026"}
Nov 02 19:47:33.055415 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:33.055291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:33.055415 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:33.055330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:33.055636 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:33.055438    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:33.055636 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:33.055532    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:33.055636 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:33.055583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:33.055809 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:33.055694    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:33.129011 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:33.128129    2578 generic.go:356] "Generic (PLEG): container finished" podID="f28b16d2fb7687a2bf2bca5db4901e34" containerID="6a595a2bb7dbd824256d77027eed46ebad0940ee2aa815380e14ac544e10577b" exitCode=0
Nov 02 19:47:33.129011 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:33.128219    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal" event={"ID":"f28b16d2fb7687a2bf2bca5db4901e34","Type":"ContainerDied","Data":"6a595a2bb7dbd824256d77027eed46ebad0940ee2aa815380e14ac544e10577b"}
Nov 02 19:47:33.135887 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:33.135467    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal" event={"ID":"09b897c9011dced250aef3fd51f6bd3f","Type":"ContainerStarted","Data":"0a9ae2fe32cb95823a3372f83628fe3d87f63e74a04b6f6ab9363a92849dbf51"}
Nov 02 19:47:33.353100 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:33.353034    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal" podStartSLOduration=7.353016917 podStartE2EDuration="7.353016917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:33.352533804 +0000 UTC m=+7.853593026" watchObservedRunningTime="2026-11-02 19:47:33.353016917 +0000 UTC m=+7.854076136"
Nov 02 19:47:34.143062 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:34.143027    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal" event={"ID":"f28b16d2fb7687a2bf2bca5db4901e34","Type":"ContainerStarted","Data":"295cc0d09b51ef64c9d7cfac865b56c288339a16dfcf92002ffa93bbed7f28f1"}
Nov 02 19:47:34.148455 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:34.148424    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jwvp7" event={"ID":"086392f7-3abd-4c9b-ae7b-69ccd1985d1b","Type":"ContainerStarted","Data":"17c0373455742ee77fd272fc88de2c0fab20eda3102bd01ff9455a3e4e2f08f4"}
Nov 02 19:47:34.256571 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:34.256511    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal" podStartSLOduration=8.256495616 podStartE2EDuration="8.256495616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:34.256052442 +0000 UTC m=+8.757111662" watchObservedRunningTime="2026-11-02 19:47:34.256495616 +0000 UTC m=+8.757555030"
Nov 02 19:47:34.285728 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:34.285695    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:47:34.286244 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:34.286222    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:47:34.482989 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:34.482884    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jwvp7" podStartSLOduration=3.866558344 podStartE2EDuration="8.482869884s" totalImagesPullingTime="4.61631154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:34.369086969 +0000 UTC m=+8.870146189" watchObservedRunningTime="2026-11-02 19:47:34.482869884 +0000 UTC m=+8.983929109"
Nov 02 19:47:34.838298 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:34.838262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:34.838481 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:34.838315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:34.838481 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:34.838465    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:34.838594 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:34.838529    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret podName:cd0732dd-9b73-4204-a64b-ba95c2aa6007 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:42.838510874 +0000 UTC m=+17.339570073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret") pod "global-pull-secret-syncer-kf87v" (UID: "cd0732dd-9b73-4204-a64b-ba95c2aa6007") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:34.838961 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:34.838935    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:34.839075 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:34.838986    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs podName:539f14e3-83e4-4288-af2e-6dbd929a1565 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:42.838972832 +0000 UTC m=+17.340032032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs") pod "network-metrics-daemon-qcr2p" (UID: "539f14e3-83e4-4288-af2e-6dbd929a1565") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:34.938835 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:34.938805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmbv\" (UniqueName: \"kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv\") pod \"network-check-target-hxstf\" (UID: \"256ea3f6-c170-4734-a39d-63fcc9049b83\") " pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:34.938957 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:34.938944    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:47:34.939011 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:34.938966    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:47:34.939011 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:34.938978    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cmbv for pod openshift-network-diagnostics/network-check-target-hxstf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:34.939098 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:34.939033    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv podName:256ea3f6-c170-4734-a39d-63fcc9049b83 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:42.939013154 +0000 UTC m=+17.440072364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmbv" (UniqueName: "kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv") pod "network-check-target-hxstf" (UID: "256ea3f6-c170-4734-a39d-63fcc9049b83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:35.055606 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:35.055574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:35.055799 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:35.055712    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:35.055869 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:35.055837    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:35.055950 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:35.055923    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:35.056020 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:35.055985    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:35.056078 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:35.056059    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:35.153828 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:35.153285    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qxbct" event={"ID":"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89","Type":"ContainerStarted","Data":"0fc9cd65f6e954a42da83c4ea8a4e16991d448bc3d9ba365bd2b789c9ad59102"}
Nov 02 19:47:35.158302 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:35.158273    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k" event={"ID":"857aedf0-97a2-469a-b0fd-765845566e58","Type":"ContainerStarted","Data":"6c337186f28104d72764af7c10b71ffa8e4ef47e066c1c3f23aa240d1cc4d9f1"}
Nov 02 19:47:35.647332 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:35.647293    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:47:35.648062 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:35.648029    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:47:35.763488 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:35.763441    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qxbct" podStartSLOduration=3.576958132 podStartE2EDuration="9.763423984s" totalImagesPullingTime="6.186465852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:35.266315447 +0000 UTC m=+9.767374667" watchObservedRunningTime="2026-11-02 19:47:35.763423984 +0000 UTC m=+10.264483195"
Nov 02 19:47:37.059736 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:37.059201    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:37.059736 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:37.059326    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:37.059736 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:37.059413    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:37.059736 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:37.059497    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:37.059736 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:37.059552    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:37.059736 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:37.059639    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:37.168575 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:37.166235    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rsl26" event={"ID":"3bd5801e-0707-4269-aef3-c22008c52025","Type":"ContainerStarted","Data":"4aa98fd34767b7d58362144619efc9ca00f2d9a629529d588269a5b71de3e673"}
Nov 02 19:47:37.170375 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:37.170346    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h6nfx" event={"ID":"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f","Type":"ContainerStarted","Data":"bc2a5340808e274c44034a34be42f45cc703fb454681f1117b6a4c9d4f302cd8"}
Nov 02 19:47:37.175251 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:37.175220    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6b6p" event={"ID":"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63","Type":"ContainerStarted","Data":"c2eabe2e7341b000bb8f03480e0ec6d3432e2c513f7eb9cba348c2ab9788e9d3"}
Nov 02 19:47:37.181101 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:37.181074    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 19:47:37.286990 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:37.286707    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rsl26" podStartSLOduration=2.656489192 podStartE2EDuration="11.286687649s" totalImagesPullingTime="8.630198457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:37.286097 +0000 UTC m=+11.787156232" watchObservedRunningTime="2026-11-02 19:47:37.286687649 +0000 UTC m=+11.787746866"
Nov 02 19:47:37.515500 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:37.515433    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-h6nfx" podStartSLOduration=2.345167454 podStartE2EDuration="11.515413055s" totalImagesPullingTime="9.170245601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:37.515379121 +0000 UTC m=+12.016438341" watchObservedRunningTime="2026-11-02 19:47:37.515413055 +0000 UTC m=+12.016472260"
Nov 02 19:47:37.916802 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:37.916687    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:47:37.181095443Z","UUID":"1857c2a7-ab74-493e-ace5-9cee96b1afae","Handler":null,"Name":"","Endpoint":""}
Nov 02 19:47:37.921476 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:37.919819    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 19:47:37.921476 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:37.919874    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 19:47:38.186996 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:38.186908    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" containerID="c2eabe2e7341b000bb8f03480e0ec6d3432e2c513f7eb9cba348c2ab9788e9d3" exitCode=0
Nov 02 19:47:38.187459 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:38.186998    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6b6p" event={"ID":"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63","Type":"ContainerDied","Data":"c2eabe2e7341b000bb8f03480e0ec6d3432e2c513f7eb9cba348c2ab9788e9d3"}
Nov 02 19:47:38.190651 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:38.190327    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k" event={"ID":"857aedf0-97a2-469a-b0fd-765845566e58","Type":"ContainerStarted","Data":"02a33f6fe1b1e870c90163084925ec4c30d88b9b0ec254fcb72b627a07be3950"}
Nov 02 19:47:39.055325 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:39.055298    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:39.055565 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:39.055331    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:39.055565 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:39.055393    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:39.055565 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:39.055406    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:39.055565 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:39.055468    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:39.055722 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:39.055581    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:39.194012 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:39.193934    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k46lq" event={"ID":"3af20298-29d1-41e2-8714-dbd4367aa89c","Type":"ContainerStarted","Data":"3b0071a6d6d24ade5c5bf725ee080d94cfb9e1461b9f9a3f009cb924cd542f4b"}
Nov 02 19:47:39.195754 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:39.195721    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mm2bm" event={"ID":"9201398f-e237-4997-9091-ffe08b8291ad","Type":"ContainerStarted","Data":"d823c686baa038b2cfc7311c01d1951ae1994a7f1f92fef08732cd1f4128e7d4"}
Nov 02 19:47:39.307290 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:39.307090    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k46lq" podStartSLOduration=4.357539627 podStartE2EDuration="13.30707046s" totalImagesPullingTime="8.949530833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:39.306807918 +0000 UTC m=+13.807867138" watchObservedRunningTime="2026-11-02 19:47:39.30707046 +0000 UTC m=+13.808129682"
Nov 02 19:47:39.420748 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:39.420689    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mm2bm" podStartSLOduration=2.14742849 podStartE2EDuration="13.420669365s" totalImagesPullingTime="11.273240875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:39.420380811 +0000 UTC m=+13.921440030" watchObservedRunningTime="2026-11-02 19:47:39.420669365 +0000 UTC m=+13.921728585"
Nov 02 19:47:40.204186 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:40.204146    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" containerID="1dee6805dc6a3e74de21dcc51d657a6e352077001e95e8805d56c98719fe569a" exitCode=0
Nov 02 19:47:40.204809 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:40.204626    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6b6p" event={"ID":"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63","Type":"ContainerDied","Data":"1dee6805dc6a3e74de21dcc51d657a6e352077001e95e8805d56c98719fe569a"}
Nov 02 19:47:40.209790 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:40.209725    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k" event={"ID":"857aedf0-97a2-469a-b0fd-765845566e58","Type":"ContainerStarted","Data":"82681e5ffbaa8686b39f0d426b6ec2bafa5e4bbac3e79cb5f8126d931de58975"}
Nov 02 19:47:40.488812 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:40.488697    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k" podStartSLOduration=4.078804284 podStartE2EDuration="14.488681472s" totalImagesPullingTime="10.409877188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:40.488039635 +0000 UTC m=+14.989098858" watchObservedRunningTime="2026-11-02 19:47:40.488681472 +0000 UTC m=+14.989740691"
Nov 02 19:47:41.056099 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:41.056061    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:41.056264 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:41.056114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:41.056264 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:41.056069    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:41.056264 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:41.056196    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:41.056418 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:41.056286    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:41.056418 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:41.056359    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:42.216574 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:42.216376    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" containerID="d4085aebd423166d7c9e333539e1b410830f60e0606cc3a552eef30640126bd5" exitCode=0
Nov 02 19:47:42.217176 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:42.216473    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6b6p" event={"ID":"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63","Type":"ContainerDied","Data":"d4085aebd423166d7c9e333539e1b410830f60e0606cc3a552eef30640126bd5"}
Nov 02 19:47:42.907042 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:42.907001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:42.907244 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:42.907049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:42.907244 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:42.907171    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:42.907244 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:42.907194    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:42.907244 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:42.907239    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs podName:539f14e3-83e4-4288-af2e-6dbd929a1565 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:58.907221362 +0000 UTC m=+33.408280584 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs") pod "network-metrics-daemon-qcr2p" (UID: "539f14e3-83e4-4288-af2e-6dbd929a1565") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:42.907399 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:42.907254    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret podName:cd0732dd-9b73-4204-a64b-ba95c2aa6007 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:58.907247583 +0000 UTC m=+33.408306780 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret") pod "global-pull-secret-syncer-kf87v" (UID: "cd0732dd-9b73-4204-a64b-ba95c2aa6007") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:43.007785 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:43.007737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmbv\" (UniqueName: \"kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv\") pod \"network-check-target-hxstf\" (UID: \"256ea3f6-c170-4734-a39d-63fcc9049b83\") " pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:43.007950 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:43.007919    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:47:43.007950 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:43.007943    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:47:43.008033 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:43.007957    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cmbv for pod openshift-network-diagnostics/network-check-target-hxstf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:43.008033 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:43.008027    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv podName:256ea3f6-c170-4734-a39d-63fcc9049b83 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:59.008004314 +0000 UTC m=+33.509063531 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmbv" (UniqueName: "kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv") pod "network-check-target-hxstf" (UID: "256ea3f6-c170-4734-a39d-63fcc9049b83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:43.055720 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:43.055686    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:43.055892 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:43.055736    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:43.055892 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:43.055698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:43.055892 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:43.055814    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:43.056011 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:43.055890    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:43.056011 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:43.055963    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:44.224621 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:44.224440    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6b6p" event={"ID":"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63","Type":"ContainerStarted","Data":"7de362dcd97dd79b42cdfdaf818217c7eb9998abf6d9679b8c2e47135f217019"}
Nov 02 19:47:44.227682 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:44.227660    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t" event={"ID":"9c982442-0aef-45b4-acd0-16e707422ca9","Type":"ContainerStarted","Data":"bb7416abb1f043b48d05f267c7662301ea1b69e32f0e8e79d234170cbea6b69c"}
Nov 02 19:47:44.227790 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:44.227722    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t" event={"ID":"9c982442-0aef-45b4-acd0-16e707422ca9","Type":"ContainerStarted","Data":"b55acdd80c4fa9b42aa7d12143af8b5ea79d01a6cc6e276ab9b18040eab944f7"}
Nov 02 19:47:44.227790 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:44.227752    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t" event={"ID":"9c982442-0aef-45b4-acd0-16e707422ca9","Type":"ContainerStarted","Data":"c297132bdca5bd1aded187e694d439d950f28ac8e4dcdac6bf81a75739bc6830"}
Nov 02 19:47:44.227865 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:44.227804    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t" event={"ID":"9c982442-0aef-45b4-acd0-16e707422ca9","Type":"ContainerStarted","Data":"41834a1d102c95d954f484546482877e7a82bbe55f79c3743d0954bf0273bfe4"}
Nov 02 19:47:44.227865 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:44.227833    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t" event={"ID":"9c982442-0aef-45b4-acd0-16e707422ca9","Type":"ContainerStarted","Data":"9349e9a4b3beda181cf31f3a0042c50afa6c2c85e72e5796030f0cf9195b72ab"}
Nov 02 19:47:44.227923 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:44.227864    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t" event={"ID":"9c982442-0aef-45b4-acd0-16e707422ca9","Type":"ContainerStarted","Data":"8cd1dbeb0587187ea0b584f11e40d164ea3125f97b850a759ff712e4218f6ccb"}
Nov 02 19:47:45.055763 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:45.055728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:45.055956 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:45.055785    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:45.055956 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:45.055842    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:45.055956 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:45.055893    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:45.056124 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:45.055961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:45.056124 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:45.056075    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:45.232804 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:45.232760    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" containerID="7de362dcd97dd79b42cdfdaf818217c7eb9998abf6d9679b8c2e47135f217019" exitCode=0
Nov 02 19:47:45.233196 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:45.232838    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6b6p" event={"ID":"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63","Type":"ContainerDied","Data":"7de362dcd97dd79b42cdfdaf818217c7eb9998abf6d9679b8c2e47135f217019"}
Nov 02 19:47:46.237501 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:46.237462    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" containerID="0ecd8bf629a8b1597ca45783c0ca33e0cdb71ca673d82b654a46bf7786ba41d9" exitCode=0
Nov 02 19:47:46.238152 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:46.237549    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6b6p" event={"ID":"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63","Type":"ContainerDied","Data":"0ecd8bf629a8b1597ca45783c0ca33e0cdb71ca673d82b654a46bf7786ba41d9"}
Nov 02 19:47:46.240886 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:46.240865    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t" event={"ID":"9c982442-0aef-45b4-acd0-16e707422ca9","Type":"ContainerStarted","Data":"f709ef95e3349b5a7cbc3f01348e1333634e75c5a27523827016e8bfdf829873"}
Nov 02 19:47:47.056110 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:47.056077    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:47.056302 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:47.056129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:47.056302 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:47.056223    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:47.056302 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:47.056221    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:47.056419 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:47.056326    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:47.056419 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:47.056376    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:47.246102 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:47.246064    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6b6p" event={"ID":"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63","Type":"ContainerStarted","Data":"8238e9df5a23201d53211af218cd5a88881cd5b2d8ce3f1b5725747f3e62c069"}
Nov 02 19:47:47.341031 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:47.340942    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:47.366022 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:47.365978    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h6b6p" podStartSLOduration=8.777270304 podStartE2EDuration="21.365967438s" totalImagesPullingTime="12.588697134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:47.36519176 +0000 UTC m=+21.866250979" watchObservedRunningTime="2026-11-02 19:47:47.365967438 +0000 UTC m=+21.867026657"
Nov 02 19:47:47.579787 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:47.579737    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:48.252450 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:48.252417    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t" event={"ID":"9c982442-0aef-45b4-acd0-16e707422ca9","Type":"ContainerStarted","Data":"17b24c9395288747a78ef599b05748f3d2ddebe9ebbfb99ac518b3de242a2da3"}
Nov 02 19:47:48.270918 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:48.270706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:48.506660 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:48.506574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:48.535689 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:48.535636    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t" podStartSLOduration=6.506251373 podStartE2EDuration="22.535622509s" totalImagesPullingTime="16.029371136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:48.535263906 +0000 UTC m=+23.036323124" watchObservedRunningTime="2026-11-02 19:47:48.535622509 +0000 UTC m=+23.036681770"
Nov 02 19:47:49.055675 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:49.055633    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:49.055938 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:49.055915    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:49.056013 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:49.055913    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:49.056087 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:49.056058    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:49.056240 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:49.056222    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:49.056750 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:49.056725    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:49.255539 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:49.255508    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:49.255998 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:49.255633    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:49.255998 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:49.255684    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:49.272971 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:49.272947    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:49.273178 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:49.273051    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:47:50.811600 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:50.811570    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qcr2p"]
Nov 02 19:47:50.812036 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:50.811693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:50.812036 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:50.811822    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:50.814341 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:50.814315    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kf87v"]
Nov 02 19:47:50.814452 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:50.814415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:50.814506 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:50.814489    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:50.861881 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:50.861855    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hxstf"]
Nov 02 19:47:50.861988 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:50.861935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:50.862020 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:50.862003    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:51.007659 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:51.007624    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:51.195974 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:51.195883    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:51.381710 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:51.381677    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:51.573183 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:51.573148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:51.756563 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:51.756533    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:51.934908 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:51.934834    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:52.055959 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:52.055929    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:52.056116 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:52.056034    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:52.405688 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:52.405656    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:52.677284 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:52.677252    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:53.055903 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:53.055875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:53.056326 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:53.055912    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:53.056326 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:53.055985    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:53.056326 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:53.056133    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:53.242161 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:53.242127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:54.058252 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:54.058226    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:54.058720 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:54.058354    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:54.201630 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:54.201595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:54.437780 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:54.437681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:54.687705 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:54.687676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:54.865605 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:54.865557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:55.055721 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:55.055687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:55.055721 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:55.055722    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:55.055962 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:55.055824    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:55.056020 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:55.055977    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:55.239053 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:55.238967    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:55.978752 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:55.978721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:56.059342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:56.059316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:56.059492 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:56.059415    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:56.061852 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:56.061835    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h6nfx_bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f/dns-node-resolver/0.log"
Nov 02 19:47:56.164398 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:56.164368    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:56.435417 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:56.435231    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:56.653987 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:56.653945    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:56.827734 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:56.827706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:57.000662 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:57.000613    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:57.056111 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:57.056083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:57.056268 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:57.056191    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:57.056268 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:57.056245    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:57.056342 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:57.056318    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:57.234294 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:57.234215    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:57.260431 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:57.260410    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qxbct_220ffe56-58a3-4bd9-b93a-dcb1ac38fc89/node-ca/0.log"
Nov 02 19:47:58.029812 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:58.029757    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:58.055391 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:58.055368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:58.055501 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:58.055478    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:47:58.483194 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:58.483114    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:58.682091 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:58.682059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:58.860451 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:58.860419    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:58.929324 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:58.929296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:47:58.929475 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:58.929335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:58.929475 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:58.929430    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:58.929475 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:58.929473    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret podName:cd0732dd-9b73-4204-a64b-ba95c2aa6007 nodeName:}" failed. No retries permitted until 2026-11-02 19:48:30.929461132 +0000 UTC m=+65.430520330 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret") pod "global-pull-secret-syncer-kf87v" (UID: "cd0732dd-9b73-4204-a64b-ba95c2aa6007") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:47:58.929577 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:58.929431    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:58.929577 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:58.929548    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs podName:539f14e3-83e4-4288-af2e-6dbd929a1565 nodeName:}" failed. No retries permitted until 2026-11-02 19:48:30.929534908 +0000 UTC m=+65.430594122 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs") pod "network-metrics-daemon-qcr2p" (UID: "539f14e3-83e4-4288-af2e-6dbd929a1565") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:47:59.030348 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:59.030313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmbv\" (UniqueName: \"kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv\") pod \"network-check-target-hxstf\" (UID: \"256ea3f6-c170-4734-a39d-63fcc9049b83\") " pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:59.030726 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:59.030450    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:47:59.030726 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:59.030463    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:47:59.030726 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:59.030474    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cmbv for pod openshift-network-diagnostics/network-check-target-hxstf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:59.030726 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:59.030519    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv podName:256ea3f6-c170-4734-a39d-63fcc9049b83 nodeName:}" failed. No retries permitted until 2026-11-02 19:48:31.03050773 +0000 UTC m=+65.531566928 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmbv" (UniqueName: "kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv") pod "network-check-target-hxstf" (UID: "256ea3f6-c170-4734-a39d-63fcc9049b83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:47:59.055651 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:59.055630    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:47:59.055783 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:59.055633    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:47:59.055824 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:59.055710    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:47:59.055824 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:47:59.055812    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:47:59.231103 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:59.231015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:47:59.446539 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:47:59.446505    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:00.025580 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:00.025546    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:00.055559 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:00.055534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:00.055916 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:00.055641    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:48:00.430343 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:00.430312    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:01.055358 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:01.055333    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:01.055526 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:01.055333    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:01.055526 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:01.055433    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:48:01.055526 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:01.055489    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:48:01.633388 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:01.633351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:01.806311 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:01.806274    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:02.032606 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:02.032570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:02.055147 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:02.055125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:02.055283 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:02.055261    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:48:02.427341 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:02.427246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:02.628810 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:02.628782    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:02.826554 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:02.826521    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:03.055174 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:03.055140    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:03.055333 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:03.055253    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:48:03.055333 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:03.055294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:03.055409 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:03.055384    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:48:03.234735 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:03.234651    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:04.052194 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:04.052157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:04.055974 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:04.055960    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:04.056074 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:04.056057    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:48:04.292758 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:04.292727    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:04.467283 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:04.467190    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:04.636909 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:04.636876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:05.055610 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:05.055575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:05.056116 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:05.055583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:05.056116 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:05.055709    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:48:05.056116 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:05.055759    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:48:06.056895 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:06.056858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:06.057382 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:06.056991    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:48:06.866737 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:06.866700    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:07.055959 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:07.055930    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:07.055959 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:07.055961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:07.056153 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:07.056037    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:48:07.056153 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:07.056142    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:48:07.152650 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:07.152568    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:08.055853 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:08.055823    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:08.056029 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:08.055956    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:48:09.055195 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:09.055165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:09.055640 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:09.055253    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:48:09.055640 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:09.055171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:09.055640 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:09.055326    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:48:10.055501 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:10.055474    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:10.055882 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:10.055576    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:48:11.055985 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:11.055951    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:11.055985 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:11.055991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:11.056462 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:11.056084    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf87v" podUID="cd0732dd-9b73-4204-a64b-ba95c2aa6007"
Nov 02 19:48:11.056462 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:11.056187    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxstf" podUID="256ea3f6-c170-4734-a39d-63fcc9049b83"
Nov 02 19:48:12.055526 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.055492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:12.055692 ip-10-0-134-196 kubenswrapper[2578]: E1102 19:48:12.055604    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qcr2p" podUID="539f14e3-83e4-4288-af2e-6dbd929a1565"
Nov 02 19:48:12.230306 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.230274    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:12.432862 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.432782    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:12.755644 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.755571    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:12.784995 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.784975    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeReady"
Nov 02 19:48:12.785116 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.785098    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 19:48:12.876778 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.876748    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pgp55"]
Nov 02 19:48:12.878462 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.878447    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:12.889600 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.889578    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-clwhz"]
Nov 02 19:48:12.891167 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.891152    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-clwhz"
Nov 02 19:48:12.899707 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.899686    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6kgdb"]
Nov 02 19:48:12.901222 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.901209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:12.934242 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.934219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:12.934349 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.934247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgwns\" (UniqueName: \"kubernetes.io/projected/143007dd-d254-458e-b203-ba223e8ff61c-kube-api-access-mgwns\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:12.934349 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.934263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-crio-socket\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:12.934442 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.934344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143007dd-d254-458e-b203-ba223e8ff61c-tmp-dir\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:12.934494 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.934444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9da3a466-b51b-46e5-850a-f40c9737a500-cert\") pod \"ingress-canary-clwhz\" (UID: \"9da3a466-b51b-46e5-850a-f40c9737a500\") " pod="openshift-ingress-canary/ingress-canary-clwhz"
Nov 02 19:48:12.934494 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.934451    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7cqmh\""
Nov 02 19:48:12.934494 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.934488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/143007dd-d254-458e-b203-ba223e8ff61c-metrics-tls\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:12.934591 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.934530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55smn\" (UniqueName: \"kubernetes.io/projected/9da3a466-b51b-46e5-850a-f40c9737a500-kube-api-access-55smn\") pod \"ingress-canary-clwhz\" (UID: \"9da3a466-b51b-46e5-850a-f40c9737a500\") " pod="openshift-ingress-canary/ingress-canary-clwhz"
Nov 02 19:48:12.934591 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.934553    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 19:48:12.934662 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.934588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvw5f\" (UniqueName: \"kubernetes.io/projected/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-kube-api-access-nvw5f\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:12.934662 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.934652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-data-volume\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:12.934743 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.934672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/143007dd-d254-458e-b203-ba223e8ff61c-config-volume\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:12.934743 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.934689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:12.942308 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.942289    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pgp55"]
Nov 02 19:48:12.946081 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.946064    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-clwhz"]
Nov 02 19:48:12.955427 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.955400    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 19:48:12.955427 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.955417    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 19:48:12.955427 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.955406    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 19:48:12.962520 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.962500    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 19:48:12.962671 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.962657    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 19:48:12.962861 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.962842    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 19:48:12.968294 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.968276    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6kgdb"]
Nov 02 19:48:12.974895 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:12.974875    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:13.034646 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.034574    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 19:48:13.034823 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.034807    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-t7wfl\""
Nov 02 19:48:13.034873 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.034850    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dsl5q\""
Nov 02 19:48:13.034873 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.034870    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 19:48:13.034956 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.034905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/143007dd-d254-458e-b203-ba223e8ff61c-config-volume\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:13.034956 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.034929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:13.035024 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.034954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:13.035024 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.034973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwns\" (UniqueName: \"kubernetes.io/projected/143007dd-d254-458e-b203-ba223e8ff61c-kube-api-access-mgwns\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:13.035024 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.034988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-crio-socket\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:13.035024 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.035007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143007dd-d254-458e-b203-ba223e8ff61c-tmp-dir\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:13.035241 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.035223    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-crio-socket\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:13.035347 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.035331    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143007dd-d254-458e-b203-ba223e8ff61c-tmp-dir\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:13.035572 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.035550    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:13.035721 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.035578    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/143007dd-d254-458e-b203-ba223e8ff61c-config-volume\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:13.035721 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.035576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9da3a466-b51b-46e5-850a-f40c9737a500-cert\") pod \"ingress-canary-clwhz\" (UID: \"9da3a466-b51b-46e5-850a-f40c9737a500\") " pod="openshift-ingress-canary/ingress-canary-clwhz"
Nov 02 19:48:13.035721 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.035632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/143007dd-d254-458e-b203-ba223e8ff61c-metrics-tls\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:13.035721 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.035668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55smn\" (UniqueName: \"kubernetes.io/projected/9da3a466-b51b-46e5-850a-f40c9737a500-kube-api-access-55smn\") pod \"ingress-canary-clwhz\" (UID: \"9da3a466-b51b-46e5-850a-f40c9737a500\") " pod="openshift-ingress-canary/ingress-canary-clwhz"
Nov 02 19:48:13.035721 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.035686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvw5f\" (UniqueName: \"kubernetes.io/projected/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-kube-api-access-nvw5f\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:13.035721 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.035718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-data-volume\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:13.036054 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.036001    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-data-volume\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:13.038824 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.038795    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:13.038824 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.038812    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/143007dd-d254-458e-b203-ba223e8ff61c-metrics-tls\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:13.038945 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.038821    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9da3a466-b51b-46e5-850a-f40c9737a500-cert\") pod \"ingress-canary-clwhz\" (UID: \"9da3a466-b51b-46e5-850a-f40c9737a500\") " pod="openshift-ingress-canary/ingress-canary-clwhz"
Nov 02 19:48:13.056058 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.056040    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:13.056137 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.056043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:13.091595 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.091569    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwns\" (UniqueName: \"kubernetes.io/projected/143007dd-d254-458e-b203-ba223e8ff61c-kube-api-access-mgwns\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:13.092057 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.092037    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvw5f\" (UniqueName: \"kubernetes.io/projected/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-kube-api-access-nvw5f\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:13.092199 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.092184    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55smn\" (UniqueName: \"kubernetes.io/projected/9da3a466-b51b-46e5-850a-f40c9737a500-kube-api-access-55smn\") pod \"ingress-canary-clwhz\" (UID: \"9da3a466-b51b-46e5-850a-f40c9737a500\") " pod="openshift-ingress-canary/ingress-canary-clwhz"
Nov 02 19:48:13.112389 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.112370    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 19:48:13.112489 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.112405    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 19:48:13.135046 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.135025    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tnfsn\""
Nov 02 19:48:13.135148 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.135123    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-z7kgr\""
Nov 02 19:48:13.135196 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.135161    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 19:48:13.189417 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.189398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:13.199342 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.199324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-clwhz"
Nov 02 19:48:13.212262 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.212240    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:13.590744 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.590715    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pgp55"]
Nov 02 19:48:13.591075 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:48:13.590724    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143007dd_d254_458e_b203_ba223e8ff61c.slice/crio-4eadc601b710c9346225aa83114e7183b5ea893e685afad15f5c3cbcb15c9796 WatchSource:0}: Error finding container 4eadc601b710c9346225aa83114e7183b5ea893e685afad15f5c3cbcb15c9796: Status 404 returned error can't find the container with id 4eadc601b710c9346225aa83114e7183b5ea893e685afad15f5c3cbcb15c9796
Nov 02 19:48:13.614173 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:48:13.614142    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da3a466_b51b_46e5_850a_f40c9737a500.slice/crio-e135ca1bf41eb60bb695727d515db2e2c40c8d0be1398ee5a5c0cd9d252e32a3 WatchSource:0}: Error finding container e135ca1bf41eb60bb695727d515db2e2c40c8d0be1398ee5a5c0cd9d252e32a3: Status 404 returned error can't find the container with id e135ca1bf41eb60bb695727d515db2e2c40c8d0be1398ee5a5c0cd9d252e32a3
Nov 02 19:48:13.615469 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.615449    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-clwhz"]
Nov 02 19:48:13.617350 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:48:13.617297    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0979491_9a8b_4ec4_b3e0_59ad16c4f454.slice/crio-748b63007922f892fcd5ee7918f80c4f72172c8e4afc0d352fda0f5321221fd5 WatchSource:0}: Error finding container 748b63007922f892fcd5ee7918f80c4f72172c8e4afc0d352fda0f5321221fd5: Status 404 returned error can't find the container with id 748b63007922f892fcd5ee7918f80c4f72172c8e4afc0d352fda0f5321221fd5
Nov 02 19:48:13.618400 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:13.618378    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6kgdb"]
Nov 02 19:48:14.055933 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:14.055903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:14.108487 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:14.108458    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kkdt\""
Nov 02 19:48:14.108487 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:14.108458    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 19:48:14.307927 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:14.307842    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6kgdb" event={"ID":"f0979491-9a8b-4ec4-b3e0-59ad16c4f454","Type":"ContainerStarted","Data":"82f40599e2b9b408e80520bf8230ca6593f39084a4cf73c40ba5b3401ceb9c38"}
Nov 02 19:48:14.307927 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:14.307907    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6kgdb" event={"ID":"f0979491-9a8b-4ec4-b3e0-59ad16c4f454","Type":"ContainerStarted","Data":"748b63007922f892fcd5ee7918f80c4f72172c8e4afc0d352fda0f5321221fd5"}
Nov 02 19:48:14.308942 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:14.308912    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-clwhz" event={"ID":"9da3a466-b51b-46e5-850a-f40c9737a500","Type":"ContainerStarted","Data":"e135ca1bf41eb60bb695727d515db2e2c40c8d0be1398ee5a5c0cd9d252e32a3"}
Nov 02 19:48:14.309880 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:14.309863    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgp55" event={"ID":"143007dd-d254-458e-b203-ba223e8ff61c","Type":"ContainerStarted","Data":"4eadc601b710c9346225aa83114e7183b5ea893e685afad15f5c3cbcb15c9796"}
Nov 02 19:48:15.145743 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:15.145660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:15.314671 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:15.314633    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6kgdb" event={"ID":"f0979491-9a8b-4ec4-b3e0-59ad16c4f454","Type":"ContainerStarted","Data":"4ecb1dfc5e2a915fa9d6f22f0fc2387db58bafe459ab795fa453d3a3f260223c"}
Nov 02 19:48:16.877672 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:16.876124    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:17.321911 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:17.321878    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgp55" event={"ID":"143007dd-d254-458e-b203-ba223e8ff61c","Type":"ContainerStarted","Data":"1520788b114daec19387ed459709dd76ef67de1e52c0eebb169d0dd6670850f8"}
Nov 02 19:48:17.322085 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:17.321940    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgp55" event={"ID":"143007dd-d254-458e-b203-ba223e8ff61c","Type":"ContainerStarted","Data":"5846f4e510703359e6f3868d00fcba50089bbbc903998d873a610c43919008cd"}
Nov 02 19:48:17.322137 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:17.322103    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:18.273004 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:18.272960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:18.325243 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:18.325209    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-clwhz" event={"ID":"9da3a466-b51b-46e5-850a-f40c9737a500","Type":"ContainerStarted","Data":"5e2bbb049f5e63225e3de604a4bd8d3ae60f6a103edfe505f937817147803e26"}
Nov 02 19:48:18.327311 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:18.327289    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6kgdb" event={"ID":"f0979491-9a8b-4ec4-b3e0-59ad16c4f454","Type":"ContainerStarted","Data":"1719a634cdafd9d1e9ec24a96ab171306e4d9728863b191a5586fc8417279272"}
Nov 02 19:48:18.441629 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:18.441579    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-clwhz" podStartSLOduration=2.438113295 podStartE2EDuration="6.44156596s" totalImagesPullingTime="4.003452665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:18.441279282 +0000 UTC m=+52.942338502" watchObservedRunningTime="2026-11-02 19:48:18.44156596 +0000 UTC m=+52.942625179"
Nov 02 19:48:18.442110 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:18.442085    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-pgp55" podStartSLOduration=2.949993261 podStartE2EDuration="6.442078792s" totalImagesPullingTime="3.492085531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:17.440339935 +0000 UTC m=+51.941399154" watchObservedRunningTime="2026-11-02 19:48:18.442078792 +0000 UTC m=+52.943138061"
Nov 02 19:48:18.461650 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:18.461620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:18.560009 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:18.559910    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6kgdb" podStartSLOduration=2.957653783 podStartE2EDuration="6.559893834s" totalImagesPullingTime="3.602240051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:48:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:18.559067677 +0000 UTC m=+53.060126895" watchObservedRunningTime="2026-11-02 19:48:18.559893834 +0000 UTC m=+53.060953054"
Nov 02 19:48:18.764829 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:18.764792    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:18.989836 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:18.989723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:19.946957 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:19.946923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:20.164962 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:20.164931    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:20.334871 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:20.334837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:20.930090 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:20.930053    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:20.962067 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:20.962029    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb"]
Nov 02 19:48:20.964003 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:20.963988    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb"
Nov 02 19:48:20.997749 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:20.997714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e4c78d9b-2b7d-4522-af17-f595eae2ab56-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lbcmb\" (UID: \"e4c78d9b-2b7d-4522-af17-f595eae2ab56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb"
Nov 02 19:48:21.017707 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:21.017672    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb"]
Nov 02 19:48:21.066703 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:21.066674    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 19:48:21.067018 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:21.067000    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bv4cs\""
Nov 02 19:48:21.098185 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:21.098152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e4c78d9b-2b7d-4522-af17-f595eae2ab56-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lbcmb\" (UID: \"e4c78d9b-2b7d-4522-af17-f595eae2ab56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb"
Nov 02 19:48:21.100467 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:21.100448    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e4c78d9b-2b7d-4522-af17-f595eae2ab56-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lbcmb\" (UID: \"e4c78d9b-2b7d-4522-af17-f595eae2ab56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb"
Nov 02 19:48:21.147324 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:21.147293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:21.269068 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:21.269042    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:21.273728 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:21.273710    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb"
Nov 02 19:48:21.632264 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:48:21.632235    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c78d9b_2b7d_4522_af17_f595eae2ab56.slice/crio-403a4e41a26323ba9c4196cba03566ce84c5dfbe2123ea1d50b6551bdcd6c4db WatchSource:0}: Error finding container 403a4e41a26323ba9c4196cba03566ce84c5dfbe2123ea1d50b6551bdcd6c4db: Status 404 returned error can't find the container with id 403a4e41a26323ba9c4196cba03566ce84c5dfbe2123ea1d50b6551bdcd6c4db
Nov 02 19:48:21.633356 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:21.633335    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb"]
Nov 02 19:48:22.337761 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:22.337718    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb" event={"ID":"e4c78d9b-2b7d-4522-af17-f595eae2ab56","Type":"ContainerStarted","Data":"403a4e41a26323ba9c4196cba03566ce84c5dfbe2123ea1d50b6551bdcd6c4db"}
Nov 02 19:48:23.342824 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:23.342713    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb" event={"ID":"e4c78d9b-2b7d-4522-af17-f595eae2ab56","Type":"ContainerStarted","Data":"3197b8a34b7191b08121b73dede0a026aec60adc901b4e4c5ff52c2726bffda1"}
Nov 02 19:48:23.343288 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:23.342916    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb"
Nov 02 19:48:23.347553 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:23.347531    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb"
Nov 02 19:48:23.455462 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:23.455407    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb" podStartSLOduration=2.042700415 podStartE2EDuration="3.455394655s" totalImagesPullingTime="1.41269424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:23.454759356 +0000 UTC m=+57.955818575" watchObservedRunningTime="2026-11-02 19:48:23.455394655 +0000 UTC m=+57.956453874"
Nov 02 19:48:24.192168 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.192135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:24.406256 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.406211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:24.540453 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.540425    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"]
Nov 02 19:48:24.542445 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.542429    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:24.597854 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.597747    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"]
Nov 02 19:48:24.598146 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.598123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:24.622268 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.622238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5drg\" (UniqueName: \"kubernetes.io/projected/0ab2031f-ddc9-4e69-a860-4d326697de18-kube-api-access-n5drg\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:24.622268 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.622278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ab2031f-ddc9-4e69-a860-4d326697de18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:24.622456 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.622385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab2031f-ddc9-4e69-a860-4d326697de18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:24.622456 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.622417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab2031f-ddc9-4e69-a860-4d326697de18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:24.646822 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.646789    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 19:48:24.646822 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.646814    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 19:48:24.647050 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.646917    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 02 19:48:24.647097 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.647056    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 02 19:48:24.647097 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.647072    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 19:48:24.647174 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.647129    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wsdm2\""
Nov 02 19:48:24.723551 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.723508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab2031f-ddc9-4e69-a860-4d326697de18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:24.723551 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.723549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab2031f-ddc9-4e69-a860-4d326697de18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:24.723814 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.723626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5drg\" (UniqueName: \"kubernetes.io/projected/0ab2031f-ddc9-4e69-a860-4d326697de18-kube-api-access-n5drg\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:24.723814 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.723664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ab2031f-ddc9-4e69-a860-4d326697de18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:24.724379 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.724357    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab2031f-ddc9-4e69-a860-4d326697de18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:24.725923 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.725904    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab2031f-ddc9-4e69-a860-4d326697de18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:24.726042 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.726025    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ab2031f-ddc9-4e69-a860-4d326697de18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:24.775974 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.775940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:24.780481 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.780458    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5drg\" (UniqueName: \"kubernetes.io/projected/0ab2031f-ddc9-4e69-a860-4d326697de18-kube-api-access-n5drg\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:24.853912 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:24.853830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:25.204566 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.204491    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-cchgk"]
Nov 02 19:48:25.208843 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.208778    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:48:25.213864 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.213840    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"]
Nov 02 19:48:25.249632 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:48:25.249608    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab2031f_ddc9_4e69_a860_4d326697de18.slice/crio-8b7f16a716eef48a7d874193b6c245b5f3290240d528357f940ff4d6b8043db1 WatchSource:0}: Error finding container 8b7f16a716eef48a7d874193b6c245b5f3290240d528357f940ff4d6b8043db1: Status 404 returned error can't find the container with id 8b7f16a716eef48a7d874193b6c245b5f3290240d528357f940ff4d6b8043db1
Nov 02 19:48:25.260386 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.260364    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-cchgk"]
Nov 02 19:48:25.261796 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.261778    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 19:48:25.263295 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.263278    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 19:48:25.263368 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.263355    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-cjnzq\""
Nov 02 19:48:25.328483 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.328457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6a65e50-7411-4a9f-af62-92340babdbbd-tmp\") pod \"downloads-6dd789c8b7-cchgk\" (UID: \"e6a65e50-7411-4a9f-af62-92340babdbbd\") " pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:48:25.328577 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.328487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg7nf\" (UniqueName: \"kubernetes.io/projected/e6a65e50-7411-4a9f-af62-92340babdbbd-kube-api-access-fg7nf\") pod \"downloads-6dd789c8b7-cchgk\" (UID: \"e6a65e50-7411-4a9f-af62-92340babdbbd\") " pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:48:25.349944 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.349914    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd" event={"ID":"0ab2031f-ddc9-4e69-a860-4d326697de18","Type":"ContainerStarted","Data":"8b7f16a716eef48a7d874193b6c245b5f3290240d528357f940ff4d6b8043db1"}
Nov 02 19:48:25.428762 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.428735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6a65e50-7411-4a9f-af62-92340babdbbd-tmp\") pod \"downloads-6dd789c8b7-cchgk\" (UID: \"e6a65e50-7411-4a9f-af62-92340babdbbd\") " pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:48:25.429173 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.428791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg7nf\" (UniqueName: \"kubernetes.io/projected/e6a65e50-7411-4a9f-af62-92340babdbbd-kube-api-access-fg7nf\") pod \"downloads-6dd789c8b7-cchgk\" (UID: \"e6a65e50-7411-4a9f-af62-92340babdbbd\") " pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:48:25.429173 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.429074    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6a65e50-7411-4a9f-af62-92340babdbbd-tmp\") pod \"downloads-6dd789c8b7-cchgk\" (UID: \"e6a65e50-7411-4a9f-af62-92340babdbbd\") " pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:48:25.486477 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.486418    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg7nf\" (UniqueName: \"kubernetes.io/projected/e6a65e50-7411-4a9f-af62-92340babdbbd-kube-api-access-fg7nf\") pod \"downloads-6dd789c8b7-cchgk\" (UID: \"e6a65e50-7411-4a9f-af62-92340babdbbd\") " pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:48:25.519314 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.519292    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:48:25.927065 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.927040    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-cchgk"]
Nov 02 19:48:25.960792 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:25.960745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:26.140582 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:26.140524    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:26.353708 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:26.353666    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-cchgk" event={"ID":"e6a65e50-7411-4a9f-af62-92340babdbbd","Type":"ContainerStarted","Data":"954552e1991f9c1b382753f208c59b11637ab67b54b385e63364dd6a63b6dffe"}
Nov 02 19:48:26.860034 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:26.860002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:27.078630 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:27.078587    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:27.256573 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:27.256415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:27.329549 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:27.329519    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:28.361850 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:28.361808    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd" event={"ID":"0ab2031f-ddc9-4e69-a860-4d326697de18","Type":"ContainerStarted","Data":"87b172a7cbd1731c44f8a34f7e1a04d960ea2d4ac4e0d9f7b90633a06810392b"}
Nov 02 19:48:28.362308 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:28.361874    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd" event={"ID":"0ab2031f-ddc9-4e69-a860-4d326697de18","Type":"ContainerStarted","Data":"4807d84078abeb04a027d0944d88149f98c507bfd07c76312a400e0d1ae2a2c2"}
Nov 02 19:48:28.476665 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:28.476601    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd" podStartSLOduration=2.353064231 podStartE2EDuration="4.476583446s" totalImagesPullingTime="2.123519215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:28.475870658 +0000 UTC m=+62.976929876" watchObservedRunningTime="2026-11-02 19:48:28.476583446 +0000 UTC m=+62.977642667"
Nov 02 19:48:29.477256 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:29.477220    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:29.691507 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:29.691466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:30.126362 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:30.126321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:30.309885 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:30.309847    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:30.603799 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:30.603752    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:30.976002 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:30.975921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:30.976002 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:30.975979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:31.027735 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.027705    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 19:48:31.034535 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.034491    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:31.039679 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.039655    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:31.076587 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.076553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmbv\" (UniqueName: \"kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv\") pod \"network-check-target-hxstf\" (UID: \"256ea3f6-c170-4734-a39d-63fcc9049b83\") " pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:31.077887 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.077867    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 19:48:31.089087 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.089063    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:31.128388 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.128362    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 19:48:31.189047 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.189019    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 19:48:31.200336 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.200316    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cmbv\" (UniqueName: \"kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv\") pod \"network-check-target-hxstf\" (UID: \"256ea3f6-c170-4734-a39d-63fcc9049b83\") " pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:31.218611 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.218592    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kkdt\""
Nov 02 19:48:31.227848 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.227783    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:31.229230 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.229206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:31.404606 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.404578    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n9zfw"]
Nov 02 19:48:31.407180 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.407162    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.413593 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.413570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:31.418388 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.418305    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-z7kgr\""
Nov 02 19:48:31.422596 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.422577    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tnfsn\""
Nov 02 19:48:31.427646 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.427623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:31.431637 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.431494    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:31.459759 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.459729    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 19:48:31.485331 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.485300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-tls\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.485712 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.485355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-metrics-client-ca\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.485712 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.485386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-textfile\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.485712 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.485408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-sys\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.485712 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.485425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5fqq\" (UniqueName: \"kubernetes.io/projected/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-kube-api-access-r5fqq\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.485712 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.485463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-root\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.485712 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.485488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.485712 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.485519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-wtmp\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.485712 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.485552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.536879 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.536731    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 19:48:31.536879 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.536731    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 19:48:31.536879 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.536844    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v8hmw\""
Nov 02 19:48:31.586929 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.586897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-root\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.587086 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.586937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.587086 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.586967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-wtmp\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.587086 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.586999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.587086 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.587019    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-root\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.587086 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.587030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-tls\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.587339 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.587092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-metrics-client-ca\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.587339 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.587121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-textfile\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.587339 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.587151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-sys\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.587339 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.587176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5fqq\" (UniqueName: \"kubernetes.io/projected/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-kube-api-access-r5fqq\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.587595 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.587541    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-sys\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.587595 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.587542    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-wtmp\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.587803 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.587732    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-metrics-client-ca\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.587803 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.587738    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-textfile\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.588077 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.588043    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.590276 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.590257    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.590356 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.590338    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-tls\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.592255 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.592232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:31.594114 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:48:31.594088    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539f14e3_83e4_4288_af2e_6dbd929a1565.slice/crio-0dc134372cd4cdb2efe32fe2c7f08f5b7c1460108ee27fd0e9ba4f67e35f6a50 WatchSource:0}: Error finding container 0dc134372cd4cdb2efe32fe2c7f08f5b7c1460108ee27fd0e9ba4f67e35f6a50: Status 404 returned error can't find the container with id 0dc134372cd4cdb2efe32fe2c7f08f5b7c1460108ee27fd0e9ba4f67e35f6a50
Nov 02 19:48:31.594835 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.594814    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qcr2p"]
Nov 02 19:48:31.644587 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.644553    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5fqq\" (UniqueName: \"kubernetes.io/projected/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-kube-api-access-r5fqq\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.723253 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.723216    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:31.734389 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:48:31.734338    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5b4895_f27a_4de9_a084_4b6bf0a4af9d.slice/crio-ef6c4f1badc28ae20a6cde0d90c6e8ee73427240e30578b747e0395cb8e54132 WatchSource:0}: Error finding container ef6c4f1badc28ae20a6cde0d90c6e8ee73427240e30578b747e0395cb8e54132: Status 404 returned error can't find the container with id ef6c4f1badc28ae20a6cde0d90c6e8ee73427240e30578b747e0395cb8e54132
Nov 02 19:48:31.798290 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.798263    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hxstf"]
Nov 02 19:48:31.799510 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:48:31.799484    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256ea3f6_c170_4734_a39d_63fcc9049b83.slice/crio-521ee212e542f593fb1622b8355b54082b422ad7c524c3bc66a8d94f12f9cc5a WatchSource:0}: Error finding container 521ee212e542f593fb1622b8355b54082b422ad7c524c3bc66a8d94f12f9cc5a: Status 404 returned error can't find the container with id 521ee212e542f593fb1622b8355b54082b422ad7c524c3bc66a8d94f12f9cc5a
Nov 02 19:48:31.803813 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:48:31.803505    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0732dd_9b73_4204_a64b_ba95c2aa6007.slice/crio-1d77e4dabcda90930f126f41e4f6b17739e2454458f93a5e6ebd0d52b8a70e8b WatchSource:0}: Error finding container 1d77e4dabcda90930f126f41e4f6b17739e2454458f93a5e6ebd0d52b8a70e8b: Status 404 returned error can't find the container with id 1d77e4dabcda90930f126f41e4f6b17739e2454458f93a5e6ebd0d52b8a70e8b
Nov 02 19:48:31.804576 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.804542    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kf87v"]
Nov 02 19:48:31.896233 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:31.896139    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:32.097969 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:32.097925    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:32.380652 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:32.380573    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9zfw" event={"ID":"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d","Type":"ContainerStarted","Data":"ef6c4f1badc28ae20a6cde0d90c6e8ee73427240e30578b747e0395cb8e54132"}
Nov 02 19:48:32.381949 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:32.381917    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qcr2p" event={"ID":"539f14e3-83e4-4288-af2e-6dbd929a1565","Type":"ContainerStarted","Data":"0dc134372cd4cdb2efe32fe2c7f08f5b7c1460108ee27fd0e9ba4f67e35f6a50"}
Nov 02 19:48:32.383153 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:32.383118    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hxstf" event={"ID":"256ea3f6-c170-4734-a39d-63fcc9049b83","Type":"ContainerStarted","Data":"521ee212e542f593fb1622b8355b54082b422ad7c524c3bc66a8d94f12f9cc5a"}
Nov 02 19:48:32.384335 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:32.384317    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kf87v" event={"ID":"cd0732dd-9b73-4204-a64b-ba95c2aa6007","Type":"ContainerStarted","Data":"1d77e4dabcda90930f126f41e4f6b17739e2454458f93a5e6ebd0d52b8a70e8b"}
Nov 02 19:48:33.239278 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:33.239241    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:33.416151 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:33.416119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:34.033908 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.033108    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-794cd4c8c5-td84z"]
Nov 02 19:48:34.036470 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.035526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.108573 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.108149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-oauth-serving-cert\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.108573 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.108224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/477caeb6-d7be-439d-a2e9-ce396eadb00b-tmp\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.108573 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.108255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-config\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.108573 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.108284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp4ls\" (UniqueName: \"kubernetes.io/projected/477caeb6-d7be-439d-a2e9-ce396eadb00b-kube-api-access-fp4ls\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.108573 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.108335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-oauth-config\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.108573 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.108362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-service-ca\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.108573 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.108404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-serving-cert\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.141815 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.141569    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 19:48:34.141815 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.141634    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 19:48:34.141815 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.141634    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 19:48:34.141815 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.141694    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 19:48:34.141815 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.141714    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-x9fws\""
Nov 02 19:48:34.141815 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.141763    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 19:48:34.142664 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.142639    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794cd4c8c5-td84z"]
Nov 02 19:48:34.209812 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.208725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-oauth-serving-cert\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.209812 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.208797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/477caeb6-d7be-439d-a2e9-ce396eadb00b-tmp\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.209812 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.208816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-config\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.209812 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.208841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp4ls\" (UniqueName: \"kubernetes.io/projected/477caeb6-d7be-439d-a2e9-ce396eadb00b-kube-api-access-fp4ls\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.209812 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.208885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-oauth-config\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.209812 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.208906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-service-ca\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.209812 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.208941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-serving-cert\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.209812 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.209738    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/477caeb6-d7be-439d-a2e9-ce396eadb00b-tmp\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.210495 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.210324    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-config\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.210495 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.210355    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-service-ca\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.210966 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.210921    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-oauth-serving-cert\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.212365 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.212321    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-oauth-config\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.215673 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.215641    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-serving-cert\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.267789 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.267743    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp4ls\" (UniqueName: \"kubernetes.io/projected/477caeb6-d7be-439d-a2e9-ce396eadb00b-kube-api-access-fp4ls\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.349333 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.349255    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:34.400037 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.400004    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb5b4895-f27a-4de9-a084-4b6bf0a4af9d" containerID="70702ce38e7a17eccf2cb521663415784e3332ef39981c0ccb9eaa54db5f6a46" exitCode=0
Nov 02 19:48:34.400362 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.400265    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9zfw" event={"ID":"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d","Type":"ContainerDied","Data":"70702ce38e7a17eccf2cb521663415784e3332ef39981c0ccb9eaa54db5f6a46"}
Nov 02 19:48:34.417039 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.417005    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qcr2p" event={"ID":"539f14e3-83e4-4288-af2e-6dbd929a1565","Type":"ContainerStarted","Data":"5cb7ffb42491fe8c9ad565c1db8e42fca7a8ddb3bbd0f48f60c1d7a48f61fe27"}
Nov 02 19:48:34.419865 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.417180    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qcr2p" event={"ID":"539f14e3-83e4-4288-af2e-6dbd929a1565","Type":"ContainerStarted","Data":"c5ffe4c3d6754fb778a353befa49ade122e317bc9aeb3bfadb6750b50fb3712e"}
Nov 02 19:48:34.634495 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.634393    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qcr2p" podStartSLOduration=66.854942133 podStartE2EDuration="1m8.634371972s" totalImagesPullingTime="1.779429839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:34.633675587 +0000 UTC m=+69.134734805" watchObservedRunningTime="2026-11-02 19:48:34.634371972 +0000 UTC m=+69.135431194"
Nov 02 19:48:34.739956 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:34.739917    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794cd4c8c5-td84z"]
Nov 02 19:48:34.755917 ip-10-0-134-196 kubenswrapper[2578]: W1102 19:48:34.755873    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477caeb6_d7be_439d_a2e9_ce396eadb00b.slice/crio-6c0ad274a9c56c5e2e0c307ed7fbb52181e229fd94bd975fbc869cf14894cc4b WatchSource:0}: Error finding container 6c0ad274a9c56c5e2e0c307ed7fbb52181e229fd94bd975fbc869cf14894cc4b: Status 404 returned error can't find the container with id 6c0ad274a9c56c5e2e0c307ed7fbb52181e229fd94bd975fbc869cf14894cc4b
Nov 02 19:48:35.423911 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:35.423879    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9zfw" event={"ID":"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d","Type":"ContainerStarted","Data":"cddc24bb5adfe78c324f2a216f3eee45f8f92979034dc240710faf07e17bf69c"}
Nov 02 19:48:35.424313 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:35.423938    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9zfw" event={"ID":"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d","Type":"ContainerStarted","Data":"4b1754c553269ac96062cc188b8bd7fe413f9d38f7993dc81e12017897c6c11c"}
Nov 02 19:48:35.427056 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:35.427031    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794cd4c8c5-td84z" event={"ID":"477caeb6-d7be-439d-a2e9-ce396eadb00b","Type":"ContainerStarted","Data":"6c0ad274a9c56c5e2e0c307ed7fbb52181e229fd94bd975fbc869cf14894cc4b"}
Nov 02 19:48:35.543707 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:35.543645    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n9zfw" podStartSLOduration=2.7579602530000003 podStartE2EDuration="4.543623054s" totalImagesPullingTime="1.698798997s" totalInitContainerRuntime="86.863804ms" isStatefulPod=true podCreationTimestamp="2026-11-02 19:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:35.542215664 +0000 UTC m=+70.043274883" watchObservedRunningTime="2026-11-02 19:48:35.543623054 +0000 UTC m=+70.044682275"
Nov 02 19:48:36.322919 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:36.322884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:36.618085 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:36.618041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:36.791493 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:36.791452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:37.437453 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:37.436434    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hxstf" event={"ID":"256ea3f6-c170-4734-a39d-63fcc9049b83","Type":"ContainerStarted","Data":"169b2f5cd647e6492a2aac271c4f11ee0ead2017ea79470dac64d1b25d9d7458"}
Nov 02 19:48:37.437453 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:37.437396    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:37.550930 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:37.550879    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hxstf" podStartSLOduration=66.794683207 podStartE2EDuration="1m11.550865s" totalImagesPullingTime="4.756181793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:37.549825947 +0000 UTC m=+72.050885167" watchObservedRunningTime="2026-11-02 19:48:37.550865 +0000 UTC m=+72.051924221"
Nov 02 19:48:37.736760 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:37.736671    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:37.912090 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:37.911995    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:38.087322 ip-10-0-134-196 kubenswrapper[2578]: I1102 19:48:38.087280    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:38.931176 ip-10-0-134-196 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 19:48:38.946233 ip-10-0-134-196 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 19:48:38.946441 ip-10-0-134-196 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 19:48:38.946505 ip-10-0-134-196 systemd[1]: kubelet.service: Consumed 3.826s CPU time.
Nov 02 19:48:38.954478 ip-10-0-134-196 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 19:48:39.041719 ip-10-0-134-196 kubenswrapper[5224]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:48:39.042429 ip-10-0-134-196 kubenswrapper[5224]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 19:48:39.042502 ip-10-0-134-196 kubenswrapper[5224]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:48:39.042571 ip-10-0-134-196 kubenswrapper[5224]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:48:39.045713 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045618    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:48:39.045713 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045709    5224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:48:39.045713 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045714    5224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:48:39.045713 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045719    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:48:39.045972 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045929    5224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:48:39.045972 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045942    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:48:39.045972 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045950    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:48:39.045972 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045955    5224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:48:39.045972 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045959    5224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:48:39.045972 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045964    5224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:48:39.045972 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045967    5224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:48:39.045972 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045972    5224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:48:39.045972 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045976    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045980    5224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045985    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.045998    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046003    5224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046008    5224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046014    5224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046018    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046023    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046027    5224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046031    5224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046035    5224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046041    5224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046045    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046049    5224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046053    5224 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046060    5224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046066    5224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046070    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046074    5224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:48:39.046383 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046079    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046083    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046087    5224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046093    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046098    5224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046103    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046108    5224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046112    5224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046117    5224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046121    5224 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046126    5224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046131    5224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046135    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046140    5224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046144    5224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046149    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046153    5224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046158    5224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046162    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046167    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:48:39.047259 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046171    5224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046176    5224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046180    5224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046185    5224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046189    5224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046193    5224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046197    5224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046203    5224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046207    5224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046211    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046215    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046220    5224 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046224    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046229    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046233    5224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046238    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046243    5224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046247    5224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046252    5224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046256    5224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:48:39.048094 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046260    5224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046266    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046272    5224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046277    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046281    5224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046286    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046290    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046294    5224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046299    5224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046303    5224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046308    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046312    5224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046319    5224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046326    5224 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046331    5224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046335    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046340    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046344    5224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046348    5224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:48:39.048993 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046353    5224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046359    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046364    5224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046368    5224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046372    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046377    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046381    5224 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046385    5224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046389    5224 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046394    5224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046400    5224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046404    5224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046408    5224 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046413    5224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.046419    5224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048198    5224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048257    5224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048262    5224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048268    5224 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048273    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:48:39.049814 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048279    5224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048284    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048290    5224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048296    5224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048306    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048311    5224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048315    5224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048320    5224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048324    5224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048329    5224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048334    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048339    5224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048345    5224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048349    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048354    5224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048359    5224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048363    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048373    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048378    5224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:48:39.050639 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048383    5224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048387    5224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048392    5224 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048396    5224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048405    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048410    5224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048416    5224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048420    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048425    5224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048429    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048436    5224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048446    5224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048452    5224 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048457    5224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048462    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048468    5224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048473    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048480    5224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048488    5224 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:48:39.051528 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048493    5224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048498    5224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048503    5224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048512    5224 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048517    5224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048522    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048526    5224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048531    5224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048535    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048540    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048544    5224 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048549    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048553    5224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048558    5224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048563    5224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048568    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048577    5224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048582    5224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048587    5224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048592    5224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:48:39.052369 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048597    5224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048602    5224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048606    5224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048611    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048616    5224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048620    5224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048625    5224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048629    5224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048639    5224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048644    5224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048649    5224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048653    5224 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048658    5224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048662    5224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048667    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048671    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048676    5224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048681    5224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048685    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:48:39.053275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048690    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048694    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048703    5224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048710    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048715    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048720    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048725    5224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048730    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048735    5224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048739    5224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048744    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048748    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048754    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048758    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048785    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048790    5224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048795    5224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048800    5224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048805    5224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:48:39.054188 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048809    5224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048814    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048819    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048823    5224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.048827    5224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051145    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051164    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051169    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051174    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051179    5224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051184    5224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051193    5224 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051198    5224 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051203    5224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051208    5224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051213    5224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051217    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051222    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051227    5224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051232    5224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:48:39.055024 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051236    5224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051241    5224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051246    5224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051250    5224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051259    5224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051272    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051280    5224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051285    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051290    5224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051295    5224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051300    5224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051305    5224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051311    5224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051317    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051323    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051327    5224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051332    5224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051342    5224 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051347    5224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:48:39.055722 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051351    5224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051356    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051360    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051365    5224 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051370    5224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051374    5224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051378    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051383    5224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051388    5224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051395    5224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051403    5224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051408    5224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051412    5224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051416    5224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051421    5224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051425    5224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051429    5224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051434    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051439    5224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051453    5224 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:48:39.056413 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051458    5224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051462    5224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051467    5224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051476    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051480    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051485    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051489    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051494    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051499    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051503    5224 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051508    5224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051512    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051517    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051521    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051526    5224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051530    5224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051539    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051543    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051548    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051555    5224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:48:39.057101 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051560    5224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051565    5224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051570    5224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051575    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051580    5224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051584    5224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051589    5224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051598    5224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051603    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051607    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051612    5224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051617    5224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051629    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051634    5224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051638    5224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051643    5224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051647    5224 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051652    5224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051656    5224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:48:39.057749 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051661    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051669    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051674    5224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051679    5224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051683    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051688    5224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051692    5224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051696    5224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051701    5224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051705    5224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051711    5224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051715    5224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.051722    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.051918    5224 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.051930    5224 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.051940    5224 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.051947    5224 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.051954    5224 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.051966    5224 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.051973    5224 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 19:48:39.058351 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.051980    5224 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.051986    5224 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.051991    5224 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.051997    5224 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052003    5224 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052009    5224 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052027    5224 flags.go:64] FLAG: --cgroup-root=""
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052032    5224 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052037    5224 flags.go:64] FLAG: --client-ca-file=""
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052043    5224 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052048    5224 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052059    5224 flags.go:64] FLAG: --cluster-domain=""
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052063    5224 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052069    5224 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052075    5224 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052085    5224 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052092    5224 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052097    5224 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052103    5224 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052109    5224 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052114    5224 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052120    5224 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052125    5224 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052135    5224 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 19:48:39.059074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052141    5224 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052148    5224 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052154    5224 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052159    5224 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052164    5224 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052169    5224 flags.go:64] FLAG: --enable-server="true"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052174    5224 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052189    5224 flags.go:64] FLAG: --event-burst="100"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052194    5224 flags.go:64] FLAG: --event-qps="50"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052199    5224 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052204    5224 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052210    5224 flags.go:64] FLAG: --eviction-hard=""
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052217    5224 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052222    5224 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052227    5224 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052242    5224 flags.go:64] FLAG: --eviction-soft=""
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052253    5224 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052258    5224 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052264    5224 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052269    5224 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052273    5224 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052279    5224 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052283    5224 flags.go:64] FLAG: --feature-gates=""
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052290    5224 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052300    5224 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 19:48:39.060043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052306    5224 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052311    5224 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052316    5224 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052322    5224 flags.go:64] FLAG: --help="false"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052327    5224 flags.go:64] FLAG: --hostname-override="ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052333    5224 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052338    5224 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052344    5224 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052354    5224 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052360    5224 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052364    5224 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052369    5224 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052374    5224 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052379    5224 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052384    5224 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052389    5224 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052398    5224 flags.go:64] FLAG: --kube-reserved=""
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052404    5224 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052409    5224 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052414    5224 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052419    5224 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052425    5224 flags.go:64] FLAG: --lock-file=""
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052430    5224 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052442    5224 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 19:48:39.061210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052448    5224 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052463    5224 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052468    5224 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052474    5224 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052479    5224 flags.go:64] FLAG: --logging-format="text"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052484    5224 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052489    5224 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052495    5224 flags.go:64] FLAG: --manifest-url=""
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052503    5224 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052515    5224 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052520    5224 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052530    5224 flags.go:64] FLAG: --max-pods="110"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052535    5224 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052541    5224 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052547    5224 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052552    5224 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052558    5224 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052563    5224 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052572    5224 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052585    5224 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052590    5224 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052595    5224 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052600    5224 flags.go:64] FLAG: --pod-cidr=""
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052605    5224 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052610    5224 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 19:48:39.062177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052626    5224 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052632    5224 flags.go:64] FLAG: --port="10250"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052637    5224 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052642    5224 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-02e70aca871c76228"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052647    5224 flags.go:64] FLAG: --qos-reserved=""
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052653    5224 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052658    5224 flags.go:64] FLAG: --register-node="true"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052671    5224 flags.go:64] FLAG: --register-with-taints=""
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052683    5224 flags.go:64] FLAG: --registry-burst="10"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052688    5224 flags.go:64] FLAG: --registry-qps="5"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052693    5224 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052698    5224 flags.go:64] FLAG: --reserved-memory=""
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052706    5224 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052711    5224 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052717    5224 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052724    5224 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052730    5224 flags.go:64] FLAG: --runonce="false"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052739    5224 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052744    5224 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052749    5224 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052755    5224 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052760    5224 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052783    5224 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052789    5224 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052794    5224 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052803    5224 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 19:48:39.062912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052808    5224 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052814    5224 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052819    5224 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052824    5224 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052830    5224 flags.go:64] FLAG: --system-cgroups=""
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052835    5224 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052849    5224 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052854    5224 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052859    5224 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052872    5224 flags.go:64] FLAG: --tls-min-version=""
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052877    5224 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052882    5224 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052887    5224 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052893    5224 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052909    5224 flags.go:64] FLAG: --v="2"
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052918    5224 flags.go:64] FLAG: --version="false"
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052925    5224 flags.go:64] FLAG: --vmodule=""
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052932    5224 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.052938    5224 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053434    5224 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053443    5224 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053451    5224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053456    5224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053461    5224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:48:39.063848 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053466    5224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053471    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053475    5224 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053481    5224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053485    5224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053490    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053494    5224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053499    5224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053508    5224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053527    5224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053533    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053537    5224 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053542    5224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053546    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053551    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053557    5224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053564    5224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053568    5224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053575    5224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:48:39.064560 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053580    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053589    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053594    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053598    5224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053610    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053616    5224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053621    5224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053626    5224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053630    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053635    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053641    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053646    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053651    5224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053655    5224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053664    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053669    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053673    5224 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053678    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053684    5224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:48:39.065194 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053691    5224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053697    5224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053702    5224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053706    5224 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053711    5224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053716    5224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053721    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053730    5224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053735    5224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053739    5224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053744    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053751    5224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053756    5224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053760    5224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053781    5224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053786    5224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053791    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053796    5224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053808    5224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053813    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:48:39.065775 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053822    5224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053827    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053831    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053837    5224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053842    5224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053847    5224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053852    5224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053857    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053861    5224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053866    5224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053870    5224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053875    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053879    5224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053889    5224 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053893    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053898    5224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053903    5224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053907    5224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053912    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053917    5224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:48:39.066398 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053921    5224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053926    5224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053930    5224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053937    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053942    5224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053951    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053956    5224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053961    5224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053965    5224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053970    5224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053975    5224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053986    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053991    5224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.053996    5224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054000    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054007    5224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054012    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054016    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054026    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:48:39.067141 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054031    5224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054036    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054040    5224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054045    5224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.054102    5224 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054234    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054244    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054248    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054253    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054258    5224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054263    5224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054268    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054272    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054277    5224 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054286    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:48:39.067737 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054293    5224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054303    5224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054309    5224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054315    5224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054320    5224 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054324    5224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054329    5224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054334    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054339    5224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054343    5224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054355    5224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054365    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054370    5224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054375    5224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054381    5224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054385    5224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054390    5224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054395    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054400    5224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054404    5224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:48:39.068206 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054409    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054414    5224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054418    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054422    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054431    5224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054436    5224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054442    5224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054446    5224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054451    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054455    5224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054460    5224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054465    5224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054469    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054476    5224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054481    5224 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054486    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054491    5224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054499    5224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054504    5224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054508    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:48:39.068829 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054513    5224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054518    5224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054523    5224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054537    5224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054542    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054547    5224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054553    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054558    5224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054564    5224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054574    5224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054578    5224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054583    5224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054588    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054592    5224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054597    5224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054601    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054606    5224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054610    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054615    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:48:39.069435 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054619    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054624    5224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054633    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054637    5224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054642    5224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054647    5224 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054653    5224 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054658    5224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054663    5224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054667    5224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054672    5224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054677    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054681    5224 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054686    5224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054690    5224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054700    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054704    5224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054716    5224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054721    5224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054727    5224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:48:39.070048 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054731    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054736    5224 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054740    5224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054745    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054749    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054756    5224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054762    5224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054787    5224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054792    5224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054796    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054801    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054805    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054810    5224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054815    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054822    5224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054826    5224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.054831    5224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.064630    5224 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.064651    5224 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 19:48:39.070620 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.064677    5224 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.064685    5224 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064804    5224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064813    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064818    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064824    5224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064829    5224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064835    5224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064840    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064846    5224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064851    5224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064856    5224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064862    5224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064867    5224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064872    5224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064877    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064882    5224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064886    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:48:39.071612 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064891    5224 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064896    5224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064900    5224 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064905    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064910    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064915    5224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064920    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064926    5224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064931    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064936    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064940    5224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064943    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064947    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064950    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064954    5224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064958    5224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064963    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064968    5224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064973    5224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064977    5224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:48:39.072242 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064981    5224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064986    5224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064990    5224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.064998    5224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065005    5224 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065011    5224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065016    5224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065021    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065025    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065029    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065034    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065039    5224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065045    5224 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065051    5224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065055    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065059    5224 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065064    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065069    5224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065073    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065077    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:48:39.072935 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065082    5224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065087    5224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065091    5224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065095    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065100    5224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065105    5224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065108    5224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065110    5224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065113    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065116    5224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065119    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065122    5224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065125    5224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065127    5224 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065130    5224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065134    5224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065138    5224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065142    5224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065147    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:48:39.073650 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065151    5224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065156    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065161    5224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065165    5224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065170    5224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065175    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065179    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065184    5224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065190    5224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065195    5224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065200    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065205    5224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065210    5224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065215    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065219    5224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065223    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065228    5224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065232    5224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065237    5224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:48:39.074265 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065241    5224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065246    5224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065263    5224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065268    5224 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065272    5224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065277    5224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065282    5224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065286    5224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065292    5224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065297    5224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065301    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065306    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.065352    5224 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065596    5224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065606    5224 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065611    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:48:39.074961 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065616    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065621    5224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065626    5224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065631    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065636    5224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065641    5224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065645    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065649    5224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065653    5224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065657    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065662    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065666    5224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065671    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065675    5224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065680    5224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065687    5224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065693    5224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065698    5224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065702    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:48:39.075357 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065710    5224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065714    5224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065720    5224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065724    5224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065728    5224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065732    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065737    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065741    5224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065745    5224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065750    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065755    5224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065760    5224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065779    5224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065786    5224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065791    5224 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065796    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065802    5224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065808    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065813    5224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:48:39.075828 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065818    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065824    5224 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065829    5224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065835    5224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065840    5224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065845    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065849    5224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065854    5224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065859    5224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065864    5224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065868    5224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065873    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065877    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065882    5224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065889    5224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065893    5224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065898    5224 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065902    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065907    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065911    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:48:39.076305 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065915    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065920    5224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065925    5224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065930    5224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065934    5224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065939    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065944    5224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065949    5224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065954    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065958    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065963    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065969    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065973    5224 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065979    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065984    5224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065988    5224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065993    5224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.065998    5224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066003    5224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066007    5224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:48:39.076862 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066012    5224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066016    5224 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066021    5224 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066026    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066031    5224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066036    5224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066040    5224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066046    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066050    5224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066054    5224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066059    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066063    5224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066068    5224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066073    5224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066078    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066082    5224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066087    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066092    5224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066096    5224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066101    5224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:48:39.077574 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066105    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066110    5224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066115    5224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066119    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066124    5224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.066171    5224 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066225    5224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066232    5224 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066237    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066241    5224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066246    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066250    5224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066255    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066259    5224 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066264    5224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066269    5224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:48:39.078252 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066274    5224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066277    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066281    5224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066286    5224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066291    5224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066296    5224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066300    5224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066305    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066310    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066315    5224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066319    5224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066323    5224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066328    5224 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066333    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066337    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066342    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066346    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066350    5224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066355    5224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:48:39.078669 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066360    5224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066365    5224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066369    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066374    5224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066378    5224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066383    5224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066388    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066393    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066397    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066402    5224 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066408    5224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066413    5224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066418    5224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066423    5224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066427    5224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066431    5224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066436    5224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066442    5224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066449    5224 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066454    5224 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:48:39.079256 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066459    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066465    5224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066469    5224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066473    5224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066478    5224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066482    5224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066487    5224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066492    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066496    5224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066501    5224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066505    5224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066510    5224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066515    5224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066520    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066524    5224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066529    5224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066533    5224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066537    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066541    5224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:48:39.079926 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066545    5224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066549    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066553    5224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066558    5224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066562    5224 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066566    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066571    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066575    5224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066580    5224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066585    5224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066590    5224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066594    5224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066599    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066604    5224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066609    5224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066613    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066617    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066621    5224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066625    5224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066629    5224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:48:39.080543 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066634    5224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066640    5224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066647    5224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066652    5224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066657    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066662    5224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066667    5224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066671    5224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066676    5224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066680    5224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066685    5224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066689    5224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066693    5224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066697    5224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066709    5224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066715    5224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066719    5224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:39.066726    5224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.066836    5224 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 19:48:39.081339 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.068206    5224 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 19:48:39.082115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.068290    5224 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 19:48:39.082115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.068621    5224 server.go:1014] "Starting client certificate rotation"
Nov 02 19:48:39.082115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.068795    5224 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 19:48:39.082115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.068855    5224 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:42:26 +0000 UTC" deadline="2028-04-13 01:37:50.664445054 +0000 UTC"
Nov 02 19:48:39.082115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.068911    5224 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12653h49m11.59553787s"
Nov 02 19:48:39.082115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.069508    5224 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 19:48:39.082115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.071325    5224 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 19:48:39.082115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.074064    5224 log.go:25] "Validated CRI v1 runtime API"
Nov 02 19:48:39.082115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.076276    5224 log.go:25] "Validated CRI v1 image API"
Nov 02 19:48:39.082115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.077170    5224 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 19:48:39.082115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.080483    5224 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 81737c7d-ecc7-4707-ba73-181ad48d46b2:/dev/nvme0n1p4 c20f6be1-c5e1-488e-a545-4754a5980b49:/dev/nvme0n1p3]
Nov 02 19:48:39.082577 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.080513    5224 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0014cc2b5f0c8e635f6777129ed2635b92c38a513e307a6b15f8e5b4d05c41df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0014cc2b5f0c8e635f6777129ed2635b92c38a513e307a6b15f8e5b4d05c41df/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0dc134372cd4cdb2efe32fe2c7f08f5b7c1460108ee27fd0e9ba4f67e35f6a50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0dc134372cd4cdb2efe32fe2c7f08f5b7c1460108ee27fd0e9ba4f67e35f6a50/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d77e4dabcda90930f126f41e4f6b17739e2454458f93a5e6ebd0d52b8a70e8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d77e4dabcda90930f126f41e4f6b17739e2454458f93a5e6ebd0d52b8a70e8b/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e45fb69a621bbcdb227e674c845715578ebcfb32dae085df43b92e7cb232f81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e45fb69a621bbcdb227e674c845715578ebcfb32dae085df43b92e7cb232f81/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/376af49dc92e8b997d83a2035414e5bc40a614310b1bf676d2af0767e7592e92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/376af49dc92e8b997d83a2035414e5bc40a614310b1bf676d2af0767e7592e92/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/403a4e41a26323ba9c4196cba03566ce84c5dfbe2123ea1d50b6551bdcd6c4db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/403a4e41a26323ba9c4196cba03566ce84c5dfbe2123ea1d50b6551bdcd6c4db/userdata/shm major:0 minor:251 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4eadc601b710c9346225aa83114e7183b5ea893e685afad15f5c3cbcb15c9796/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4eadc601b710c9346225aa83114e7183b5ea893e685afad15f5c3cbcb15c9796/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/521ee212e542f593fb1622b8355b54082b422ad7c524c3bc66a8d94f12f9cc5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/521ee212e542f593fb1622b8355b54082b422ad7c524c3bc66a8d94f12f9cc5a/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cd4129444488cb4bbda1fe3de110c895aabfef870d9ecf7e623276b6254b9d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cd4129444488cb4bbda1fe3de110c895aabfef870d9ecf7e623276b6254b9d5/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c0ad274a9c56c5e2e0c307ed7fbb52181e229fd94bd975fbc869cf14894cc4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c0ad274a9c56c5e2e0c307ed7fbb52181e229fd94bd975fbc869cf14894cc4b/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ca0a98c88ac99c89ccd69f236d073ecfeccf28369fa4e589b6887d032d308ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ca0a98c88ac99c89ccd69f236d073ecfeccf28369fa4e589b6887d032d308ae/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/748b63007922f892fcd5ee7918f80c4f72172c8e4afc0d352fda0f5321221fd5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/748b63007922f892fcd5ee7918f80c4f72172c8e4afc0d352fda0f5321221fd5/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b7f16a716eef48a7d874193b6c245b5f3290240d528357f940ff4d6b8043db1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b7f16a716eef48a7d874193b6c245b5f3290240d528357f940ff4d6b8043db1/userdata/shm major:0 minor:264 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/954552e1991f9c1b382753f208c59b11637ab67b54b385e63364dd6a63b6dffe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/954552e1991f9c1b382753f208c59b11637ab67b54b385e63364dd6a63b6dffe/userdata/shm major:0 minor:269 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3f556e8af1c4fbc9621ccb3f4727aeba79f2f37ee2915aee1d346972f2ddeee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3f556e8af1c4fbc9621ccb3f4727aeba79f2f37ee2915aee1d346972f2ddeee/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf2c35c34c369e2adfb593d43f1de8f796386378a20f11e0170c91e72da661db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf2c35c34c369e2adfb593d43f1de8f796386378a20f11e0170c91e72da661db/userdata/shm major:0 minor:59 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ceb2463c2c1b55a51d6e729b7c3fae2f8b825354b8b0276972ba8b29a2a534dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ceb2463c2c1b55a51d6e729b7c3fae2f8b825354b8b0276972ba8b29a2a534dc/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cef2a9f7284bde86be5b82b9bf7f686b02392a0bd023f4e1cff15bb80a174a65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cef2a9f7284bde86be5b82b9bf7f686b02392a0bd023f4e1cff15bb80a174a65/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e135ca1bf41eb60bb695727d515db2e2c40c8d0be1398ee5a5c0cd9d252e32a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e135ca1bf41eb60bb695727d515db2e2c40c8d0be1398ee5a5c0cd9d252e32a3/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef6c4f1badc28ae20a6cde0d90c6e8ee73427240e30578b747e0395cb8e54132/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef6c4f1badc28ae20a6cde0d90c6e8ee73427240e30578b747e0395cb8e54132/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f43d752c2ae6f2bf739bf9ce3c576f5bfdf1445401edd1a5576a254289b579e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f43d752c2ae6f2bf739bf9ce3c576f5bfdf1445401edd1a5576a254289b579e4/userdata/shm major:0 minor:58 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/086392f7-3abd-4c9b-ae7b-69ccd1985d1b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/086392f7-3abd-4c9b-ae7b-69ccd1985d1b/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ab2031f-ddc9-4e69-a860-4d326697de18/volumes/kubernetes.io~projected/kube-api-access-n5drg:{mountpoint:/var/lib/kubelet/pods/0ab2031f-ddc9-4e69-a860-4d326697de18/volumes/kubernetes.io~projected/kube-api-access-n5drg major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ab2031f-ddc9-4e69-a860-4d326697de18/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0ab2031f-ddc9-4e69-a860-4d326697de18/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ab2031f-ddc9-4e69-a860-4d326697de18/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/0ab2031f-ddc9-4e69-a860-4d326697de18/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/143007dd-d254-458e-b203-ba223e8ff61c/volumes/kubernetes.io~projected/kube-api-access-mgwns:{mountpoint:/var/lib/kubelet/pods/143007dd-d254-458e-b203-ba223e8ff61c/volumes/kubernetes.io~projected/kube-api-access-mgwns major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/143007dd-d254-458e-b203-ba223e8ff61c/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/143007dd-d254-458e-b203-ba223e8ff61c/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89/volumes/kubernetes.io~projected/kube-api-access-gmxqn:{mountpoint:/var/lib/kubelet/pods/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89/volumes/kubernetes.io~projected/kube-api-access-gmxqn major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/256ea3f6-c170-4734-a39d-63fcc9049b83/volumes/kubernetes.io~projected/kube-api-access-7cmbv:{mountpoint:/var/lib/kubelet/pods/256ea3f6-c170-4734-a39d-63fcc9049b83/volumes/kubernetes.io~projected/kube-api-access-7cmbv major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3af20298-29d1-41e2-8714-dbd4367aa89c/volumes/kubernetes.io~projected/kube-api-access-4pj2b:{mountpoint:/var/lib/kubelet/pods/3af20298-29d1-41e2-8714-dbd4367aa89c/volumes/kubernetes.io~projected/kube-api-access-4pj2b major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bd5801e-0707-4269-aef3-c22008c52025/volumes/kubernetes.io~projected/kube-api-access-pqgql:{mountpoint:/var/lib/kubelet/pods/3bd5801e-0707-4269-aef3-c22008c52025/volumes/kubernetes.io~projected/kube-api-access-pqgql major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/477caeb6-d7be-439d-a2e9-ce396eadb00b/volumes/kubernetes.io~projected/kube-api-access-fp4ls:{mountpoint:/var/lib/kubelet/pods/477caeb6-d7be-439d-a2e9-ce396eadb00b/volumes/kubernetes.io~projected/kube-api-access-fp4ls major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/477caeb6-d7be-439d-a2e9-ce396eadb00b/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/477caeb6-d7be-439d-a2e9-ce396eadb00b/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/477caeb6-d7be-439d-a2e9-ce396eadb00b/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/477caeb6-d7be-439d-a2e9-ce396eadb00b/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/539f14e3-83e4-4288-af2e-6dbd929a1565/volumes/kubernetes.io~projected/kube-api-access-ls4kl:{mountpoint:/var/lib/kubelet/pods/539f14e3-83e4-4288-af2e-6dbd929a1565/volumes/kubernetes.io~projected/kube-api-access-ls4kl major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/539f14e3-83e4-4288-af2e-6dbd929a1565/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/539f14e3-83e4-4288-af2e-6dbd929a1565/volumes/kubernetes.io~secret/metrics-certs major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/857aedf0-97a2-469a-b0fd-765845566e58/volumes/kubernetes.io~projected/kube-api-access-89wdg:{mountpoint:/var/lib/kubelet/pods/857aedf0-97a2-469a-b0fd-765845566e58/volumes/kubernetes.io~projected/kube-api-access-89wdg major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9201398f-e237-4997-9091-ffe08b8291ad/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9201398f-e237-4997-9091-ffe08b8291ad/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9201398f-e237-4997-9091-ffe08b8291ad/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9201398f-e237-4997-9091-ffe08b8291ad/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9201398f-e237-4997-9091-ffe08b8291ad/volumes/kubernetes.io~projected/kube-api-access-dhd5m:{mountpoint:/var/lib/kubelet/pods/9201398f-e237-4997-9091-ffe08b8291ad/volumes/kubernetes.io~projected/kube-api-access-dhd5m major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c982442-0aef-45b4-acd0-16e707422ca9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9c982442-0aef-45b4-acd0-16e707422ca9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c982442-0aef-45b4-acd0-16e707422ca9/volumes/kubernetes.io~projected/kube-api-access-knqwv:{mountpoint:/var/lib/kubelet/pods/9c982442-0aef-45b4-acd0-16e707422ca9/volumes/kubernetes.io~projected/kube-api-access-knqwv major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c982442-0aef-45b4-acd0-16e707422ca9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9c982442-0aef-45b4-acd0-16e707422ca9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9da3a466-b51b-46e5-850a-f40c9737a500/volumes/kubernetes.io~projected/kube-api-access-55smn:{mountpoint:/var/lib/kubelet/pods/9da3a466-b51b-46e5-850a-f40c9737a500/volumes/kubernetes.io~projected/kube-api-access-55smn major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9da3a466-b51b-46e5-850a-f40c9737a500/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9da3a466-b51b-46e5-850a-f40c9737a500/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f/volumes/kubernetes.io~projected/kube-api-access-zsss9:{mountpoint:/var/lib/kubelet/pods/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f/volumes/kubernetes.io~projected/kube-api-access-zsss9 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d/volumes/kubernetes.io~projected/kube-api-access-r5fqq:{mountpoint:/var/lib/kubelet/pods/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d/volumes/kubernetes.io~projected/kube-api-access-r5fqq major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd0732dd-9b73-4204-a64b-ba95c2aa6007/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/cd0732dd-9b73-4204-a64b-ba95c2aa6007/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63/volumes/kubernetes.io~projected/kube-api-access-62972:{mountpoint:/var/lib/kubelet/pods/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63/volumes/kubernetes.io~projected/kube-api-access-62972 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4c78d9b-2b7d-4522-af17-f595eae2ab56/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/e4c78d9b-2b7d-4522-af17-f595eae2ab56/volumes/kubernetes.io~secret/tls-certificates major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6a65e50-7411-4a9f-af62-92340babdbbd/volumes/kubernetes.io~projected/kube-api-access-fg7nf:{mountpoint:/var/lib/kubelet/pods/e6a65e50-7411-4a9f-af62-92340babdbbd/volumes/kubernetes.io~projected/kube-api-access-fg7nf major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0979491-9a8b-4ec4-b3e0-59ad16c4f454/volumes/kubernetes.io~projected/kube-api-access-nvw5f:{mountpoint:/var/lib/kubelet/pods/f0979491-9a8b-4ec4-b3e0-59ad16c4f454/volumes/kubernetes.io~projected/kube-api-access-nvw5f major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0979491-9a8b-4ec4-b3e0-59ad16c4f454/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/f0979491-9a8b-4ec4-b3e0-59ad16c4f454/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0d6613cdfecefb15a5175868bf0815b989c0b681ec5abc8ae49a8a2c1c414525/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6ac47973db74817402ab56fc185cee18a69a2874c96b377522557ed63df17757/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/283983172e00f435828863d3b3328cf455b33a8f0b04a75933c3cf9cd2272bcf/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9c7950714245c0442f7ba8d88c75fdc4d09a26877b181793772113001c5158a8/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/f09e01cb5260597c01bdb93b3356f5be9b0abc15d1d69330bc6e849172be53b2/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0fcfaa5ac19b336838e2ff86dfd88f6467e301ca4e9d5f2487683b0a839b1153/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5a77651244b0f1418cb9879a7a28b4df76e6422e144d61e1d84dd34c7fbef08e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1d5ecd9729c2ae3657718d77bea5c094552e0dbb5f7c79b24e1788a1fce45425/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e67bb0963614ab2946bdb47f9e969bad0b7f34b77ee6f170bf2f9592d96f8710/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/7dc916f7c253c7f45c83290f9b9623812c62af965a101a6ed7470e9b996b7ecb/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/bcb814fd3b033f82ec496d72e15c80120e0f9a4ab44c8687e0472790616b5407/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/6516c50adc10e1a7479ae6f29100327687f2e359b73e59887cb662451f8ca2a1/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e233f48d67a5d71ad072f662f7528b2325636cd0932a91cc34ffd49a603f5450/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/eec7f867c8b03474ffc0cf4ce4f1c741cc7c80ad401d7cfa2bcb7e97f5bf900f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4319bb3e955446737aa83ca13539e6f864325ebd495cad9a08df2dc1b7db19bb/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/5a6854525c44959d5887c9fd48bdd81f07e9c1cb9de1bc9457dbb64190958e5e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/5bdfc7928d9733b635892b08d5483f4f312c5c0de189c37d1b148606cd7d066a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5deb26cc91b59e971922a87d411f95aa3ec3fdfed383e7c6c3ada6eb6cce7e24/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/02dfe2408b02ab9be0dc88fd7ce5fb6a1cad79f224a1f856055ff4b2b6da420e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ce7c75b60510f11d172776bb1cb1cbfbf6523bcab2d841147bba6406e6e8513c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/05162c45a33ff27c847ab6e0eb6937ff629870889161ff2dd07bd07d8aa16b7e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/d8e6f0ab1de25958ddf581d025555027cc8286b3a15dbbdb849044e055c2b452/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/3acf937bebc362d10ade83f1513d61a16e96ff47e3e9652edac1e0270bd8e682/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/3ecf0996fd3af7525d5e8db8c37c0cf8f058cc014cabd4be987a299a2ae99a6f/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/d74c4dee4968e2e65c414624fd8ea7362a21a0cf99d4abfa7ed2fe36c3c979da/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/e2b7303f0f111812e9e94e5313206b8d5722d774126a3f407cb45e86d289eae2/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/cd7a6856fd7a47987db590a0c05309536e184aacb7d78025b49d74a2e59d36cc/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d653f674f88558c6b434e1e250166707d8a7803872c7abc0c2d6be0f978704c3/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/b37f75e6aa27598cf1398a45b377edd7ee8cfacda8f447d2d01d6ced1f271b0a/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/240d386ece1b0943da6c8fd1e983b4c17fdbfa6aa2691ef7116fd11752b6542c/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/f89527e1d2b64bc140b8d07ef53d1410d6a8bdd365a98420f54d8e706fab91cc/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/4ea03052795a7c94b6304ae255238de0a9bbd1de88a5e07248444f926c1505fc/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/8edf01be464db5dbd6e7c75d424a58c09def12c970f0229fd5ca20b51279f9f6/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/66e117620d1e6d8917e395a4f8597299c2384a17ef1830d36988275ba752a2ea/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/d63e3112959a0a88a96666a92c09aa9557388219ca9a48849b4bfed4a2ee2623/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/00313d5e4198488fce4d24771ff94cb20050e2c89ff1a788937837e1414d9eb5/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/7c0ade5657899a7870d049c5bfe3005e500dc8c151d839efd600e245c7e71c80/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/b4dc79594d7b23a61e8b2e96923fecd074e15fba9a0e75b9c8728f7f0504c065/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/c19ec1d30fdc921dc5441d01c2aa726ca5ed423393e2dde166dc8417163ca53e/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/1ca9e6dd0926fd391e4b52a12aff7bc77252cc59cea3422fff12393a0e9b1e8a/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/692ede6321e3e8a2e27158fc18959f1b51cba94389c41763cc909d67a2466808/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/f692976ca6eabf5d579ff1e6e6f5456c668e85e926dea0781ea5923d1cda083d/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/f515cd9347231786f60bec8a3f6263bd8764a91ae8214f67ed1c4943582fb755/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/cec3ff9916df1ea8f9062607d3a88a3f53bca55c405244ce9da53aef839b8b89/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/4c8c13517e87ef10dfbd23c38fd20cbf24e94a809ae72d06b7c4be475a4bc209/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/5f6d986e1e4ce9c989b0cdc3bd7fa2f8b196807eaa859160ef6d09574543c761/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/d0ef365fe2482dd6e6f6330c3cc171f339db7fd8fef9620d0a2605e2a1d4bf40/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/bd140ca9d8ba9a54d71367002ecb1e270ce582256c4f7915356ce06a26fc6f4f/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/0b3a60a5c90eec917fe6a4ea4b9bff773b26e92e09fcafcd1e942b5a8104a386/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/cb97bb85aaac372701e602786570ec3e9fdd33aea023a8fd1cc67929cbb3ea61/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/b2cc170fa8c990908afc9eb3405362706783cd16ea47f638aaa80bb3dcfdcd33/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/07306a8013785c938625fb2e1355354e90bb4ddb75bd219ce0f3f8a2b836c5a9/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c2793da27d15bf8d25f3d7feaa2745e578ca45dc98fe077133ef040e77178ce2/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/46a7c2ad3b82227e95f77a40374b07a0461546ea457e978f9b5da253e2157604/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/637024eb21c6f7d74b868208d09b1fa80ea942be04779a70be85dd981a49d977/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d7bcb1183655a59f7def11ce14910536ccb8b2a114c9367e1928c8e54f0c3747/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/09c80e0843e355952512fc825f51ea034162baff5c1ca42591cc9d3feec13077/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 19:48:39.091462 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.091019    5224 manager.go:217] Machine: {Timestamp:2026-11-02 19:48:39.090388288 +0000 UTC m=+0.088076888 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099879 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2585e84e13d597d09a69b45e10b739 SystemUUID:ec2585e8-4e13-d597-d09a-69b45e10b739 BootID:4568f2d9-d301-40c7-a35a-ea1d6b8b206b Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9201398f-e237-4997-9091-ffe08b8291ad/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4eadc601b710c9346225aa83114e7183b5ea893e685afad15f5c3cbcb15c9796/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/3af20298-29d1-41e2-8714-dbd4367aa89c/volumes/kubernetes.io~projected/kube-api-access-4pj2b DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cd4129444488cb4bbda1fe3de110c895aabfef870d9ecf7e623276b6254b9d5/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6a65e50-7411-4a9f-af62-92340babdbbd/volumes/kubernetes.io~projected/kube-api-access-fg7nf DeviceMajor:0 DeviceMinor:268 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/954552e1991f9c1b382753f208c59b11637ab67b54b385e63364dd6a63b6dffe/userdata/shm DeviceMajor:0 DeviceMinor:269 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/477caeb6-d7be-439d-a2e9-ce396eadb00b/volumes/kubernetes.io~projected/kube-api-access-fp4ls DeviceMajor:0 DeviceMinor:319 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ab2031f-ddc9-4e69-a860-4d326697de18/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:257 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63/volumes/kubernetes.io~projected/kube-api-access-62972 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/539f14e3-83e4-4288-af2e-6dbd929a1565/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:281 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9da3a466-b51b-46e5-850a-f40c9737a500/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/086392f7-3abd-4c9b-ae7b-69ccd1985d1b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4c78d9b-2b7d-4522-af17-f595eae2ab56/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f43d752c2ae6f2bf739bf9ce3c576f5bfdf1445401edd1a5576a254289b579e4/userdata/shm DeviceMajor:0 DeviceMinor:58 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/748b63007922f892fcd5ee7918f80c4f72172c8e4afc0d352fda0f5321221fd5/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/9c982442-0aef-45b4-acd0-16e707422ca9/volumes/kubernetes.io~projected/kube-api-access-knqwv DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/521ee212e542f593fb1622b8355b54082b422ad7c524c3bc66a8d94f12f9cc5a/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d77e4dabcda90930f126f41e4f6b17739e2454458f93a5e6ebd0d52b8a70e8b/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d/volumes/kubernetes.io~projected/kube-api-access-r5fqq DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3f556e8af1c4fbc9621ccb3f4727aeba79f2f37ee2915aee1d346972f2ddeee/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd0732dd-9b73-4204-a64b-ba95c2aa6007/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:285 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/256ea3f6-c170-4734-a39d-63fcc9049b83/volumes/kubernetes.io~projected/kube-api-access-7cmbv DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ab2031f-ddc9-4e69-a860-4d326697de18/volumes/kubernetes.io~projected/kube-api-access-n5drg DeviceMajor:0 DeviceMinor:263 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/539f14e3-83e4-4288-af2e-6dbd929a1565/volumes/kubernetes.io~projected/kube-api-access-ls4kl DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89/volumes/kubernetes.io~projected/kube-api-access-gmxqn DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cef2a9f7284bde86be5b82b9bf7f686b02392a0bd023f4e1cff15bb80a174a65/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9da3a466-b51b-46e5-850a-f40c9737a500/volumes/kubernetes.io~projected/kube-api-access-55smn DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/477caeb6-d7be-439d-a2e9-ce396eadb00b/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:313 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e45fb69a621bbcdb227e674c845715578ebcfb32dae085df43b92e7cb232f81/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0979491-9a8b-4ec4-b3e0-59ad16c4f454/volumes/kubernetes.io~projected/kube-api-access-nvw5f DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c0ad274a9c56c5e2e0c307ed7fbb52181e229fd94bd975fbc869cf14894cc4b/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b7f16a716eef48a7d874193b6c245b5f3290240d528357f940ff4d6b8043db1/userdata/shm DeviceMajor:0 DeviceMinor:264 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef6c4f1badc28ae20a6cde0d90c6e8ee73427240e30578b747e0395cb8e54132/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bd5801e-0707-4269-aef3-c22008c52025/volumes/kubernetes.io~projected/kube-api-access-pqgql DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ceb2463c2c1b55a51d6e729b7c3fae2f8b825354b8b0276972ba8b29a2a534dc/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0979491-9a8b-4ec4-b3e0-59ad16c4f454/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f/volumes/kubernetes.io~projected/kube-api-access-zsss9 DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e135ca1bf41eb60bb695727d515db2e2c40c8d0be1398ee5a5c0cd9d252e32a3/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/857aedf0-97a2-469a-b0fd-765845566e58/volumes/kubernetes.io~projected/kube-api-access-89wdg DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9201398f-e237-4997-9091-ffe08b8291ad/volumes/kubernetes.io~projected/kube-api-access-dhd5m DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/143007dd-d254-458e-b203-ba223e8ff61c/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0dc134372cd4cdb2efe32fe2c7f08f5b7c1460108ee27fd0e9ba4f67e35f6a50/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/143007dd-d254-458e-b203-ba223e8ff61c/volumes/kubernetes.io~projected/kube-api-access-mgwns DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/477caeb6-d7be-439d-a2e9-ce396eadb00b/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9c982442-0aef-45b4-acd0-16e707422ca9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9201398f-e237-4997-9091-ffe08b8291ad/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9c982442-0aef-45b4-acd0-16e707422ca9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:294 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/403a4e41a26323ba9c4196cba03566ce84c5dfbe2123ea1d50b6551bdcd6c4db/userdata/shm DeviceMajor:0 DeviceMinor:251 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0ab2031f-ddc9-4e69-a860-4d326697de18/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:262 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/376af49dc92e8b997d83a2035414e5bc40a614310b1bf676d2af0767e7592e92/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ca0a98c88ac99c89ccd69f236d073ecfeccf28369fa4e589b6887d032d308ae/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0014cc2b5f0c8e635f6777129ed2635b92c38a513e307a6b15f8e5b4d05c41df/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf2c35c34c369e2adfb593d43f1de8f796386378a20f11e0170c91e72da661db/userdata/shm DeviceMajor:0 DeviceMinor:59 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0dc134372cd4cdb MacAddress:9a:d2:16:bf:72:2c Speed:10000 Mtu:8901} {Name:1d77e4dabcda909 MacAddress:9a:21:29:d9:3c:25 Speed:10000 Mtu:8901} {Name:403a4e41a26323b MacAddress:d6:72:ce:55:a8:43 Speed:10000 Mtu:8901} {Name:4eadc601b710c93 MacAddress:1e:80:19:46:4e:1b Speed:10000 Mtu:8901} {Name:521ee212e542f59 MacAddress:1e:f3:57:fd:b8:75 Speed:10000 Mtu:8901} {Name:6c0ad274a9c56c5 MacAddress:56:20:2d:ed:3b:1c Speed:10000 Mtu:8901} {Name:748b63007922f89 MacAddress:7e:6d:4f:75:f7:f7 Speed:10000 Mtu:8901} {Name:8b7f16a716eef48 MacAddress:46:f7:26:21:42:3d Speed:10000 Mtu:8901} {Name:954552e1991f9c1 MacAddress:3a:5f:f4:61:93:ed Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e3:e9:e7:b1 Speed:0 Mtu:9001} {Name:br-int MacAddress:32:57:c3:6d:c3:e0 Speed:0 Mtu:8901} {Name:e135ca1bf41eb60 MacAddress:ca:9e:9a:a9:fc:24 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e3:e9:e7:b1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0a:36:99:cb:f8:4a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8e:7c:20:9e:45:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 19:48:39.091820 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.091469    5224 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 19:48:39.091820 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.091533    5224 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 19:48:39.091928 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.091878    5224 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 19:48:39.092241 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.091910    5224 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-196.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 19:48:39.092241 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.092194    5224 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 19:48:39.092241 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.092226    5224 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 19:48:39.092435 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.092256    5224 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 19:48:39.092435 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.092274    5224 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 19:48:39.092435 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.092392    5224 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 19:48:39.092618 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.092605    5224 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 19:48:39.092758 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.092737    5224 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 19:48:39.092878 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.092794    5224 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 19:48:39.092878 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.092815    5224 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 19:48:39.092878 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.092829    5224 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 19:48:39.092878 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.092844    5224 kubelet.go:405] "Adding apiserver pod source"
Nov 02 19:48:39.092878 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.092855    5224 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 19:48:39.093527 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.093513    5224 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 19:48:39.093527 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.093528    5224 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 19:48:39.094560 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.094537    5224 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 19:48:39.106748 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.106731    5224 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 19:48:39.106748 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.106746    5224 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 19:48:39.106930 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.106908    5224 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 19:48:39.107136 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107123    5224 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 19:48:39.107197 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107169    5224 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 19:48:39.107362 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107286    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 19:48:39.107362 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107313    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 19:48:39.107362 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107329    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 19:48:39.107362 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107341    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 19:48:39.107362 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107351    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 19:48:39.107600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107366    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 19:48:39.107600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107379    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 19:48:39.107600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107389    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 19:48:39.107600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107402    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 19:48:39.107600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107411    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 19:48:39.107600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107432    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 19:48:39.107600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107447    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 19:48:39.107600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107480    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 19:48:39.107600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.107488    5224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 19:48:39.112920 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.111342    5224 server.go:1263] "Started kubelet"
Nov 02 19:48:39.112920 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.112330    5224 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 19:48:39.112920 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.112480    5224 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 19:48:39.112920 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.112554    5224 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 19:48:39.117108 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.113925    5224 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 19:48:39.117108 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.116298    5224 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 19:48:39.117108 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.116726    5224 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 19:48:39.114099 ip-10-0-134-196 systemd[1]: Started Kubernetes Kubelet.
Nov 02 19:48:39.117385 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.117345    5224 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 19:48:39.117385 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.117362    5224 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 19:48:39.118257 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.117719    5224 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 19:48:39.118257 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.117796    5224 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 19:48:39.120096 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.120045    5224 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:42:27 +0000 UTC" deadline="2028-06-25 03:49:55.547404742 +0000 UTC"
Nov 02 19:48:39.120178 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.120097    5224 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14408h1m16.42731277s"
Nov 02 19:48:39.120446 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:39.120427    5224 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-196.us-west-2.compute.internal\" not found"
Nov 02 19:48:39.120984 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.120966    5224 factory.go:55] Registering systemd factory
Nov 02 19:48:39.120984 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.120985    5224 factory.go:223] Registration of the systemd container factory successfully
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.121828    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="477caeb6-d7be-439d-a2e9-ce396eadb00b" volumeName="kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-serving-cert" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.121858    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ab2031f-ddc9-4e69-a860-4d326697de18" volumeName="kubernetes.io/secret/0ab2031f-ddc9-4e69-a860-4d326697de18-prometheus-operator-tls" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.121874    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="220ffe56-58a3-4bd9-b93a-dcb1ac38fc89" volumeName="kubernetes.io/projected/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-kube-api-access-gmxqn" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.121886    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="477caeb6-d7be-439d-a2e9-ce396eadb00b" volumeName="kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-oauth-serving-cert" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.121898    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ab2031f-ddc9-4e69-a860-4d326697de18" volumeName="kubernetes.io/configmap/0ab2031f-ddc9-4e69-a860-4d326697de18-metrics-client-ca" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.121910    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ab2031f-ddc9-4e69-a860-4d326697de18" volumeName="kubernetes.io/projected/0ab2031f-ddc9-4e69-a860-4d326697de18-kube-api-access-n5drg" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.121923    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bd5801e-0707-4269-aef3-c22008c52025" volumeName="kubernetes.io/configmap/3bd5801e-0707-4269-aef3-c22008c52025-multus-daemon-config" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.121935    5224 factory.go:153] Registering CRI-O factory
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.121948    5224 factory.go:223] Registration of the crio container factory successfully
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.121935    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0979491-9a8b-4ec4-b3e0-59ad16c4f454" volumeName="kubernetes.io/configmap/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.121969    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb5b4895-f27a-4de9-a084-4b6bf0a4af9d" volumeName="kubernetes.io/empty-dir/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-textfile" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.121981    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="539f14e3-83e4-4288-af2e-6dbd929a1565" volumeName="kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.121993    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" volumeName="kubernetes.io/projected/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-kube-api-access-62972" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122001    5224 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122005    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6a65e50-7411-4a9f-af62-92340babdbbd" volumeName="kubernetes.io/projected/e6a65e50-7411-4a9f-af62-92340babdbbd-kube-api-access-fg7nf" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122018    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="143007dd-d254-458e-b203-ba223e8ff61c" volumeName="kubernetes.io/configmap/143007dd-d254-458e-b203-ba223e8ff61c-config-volume" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122030    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af20298-29d1-41e2-8714-dbd4367aa89c" volumeName="kubernetes.io/projected/3af20298-29d1-41e2-8714-dbd4367aa89c-kube-api-access-4pj2b" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122049    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f" volumeName="kubernetes.io/projected/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-kube-api-access-zsss9" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122060    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb5b4895-f27a-4de9-a084-4b6bf0a4af9d" volumeName="kubernetes.io/configmap/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-metrics-client-ca" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122080    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb5b4895-f27a-4de9-a084-4b6bf0a4af9d" volumeName="kubernetes.io/projected/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-kube-api-access-r5fqq" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122093    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd0732dd-9b73-4204-a64b-ba95c2aa6007" volumeName="kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122106    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c982442-0aef-45b4-acd0-16e707422ca9" volumeName="kubernetes.io/projected/9c982442-0aef-45b4-acd0-16e707422ca9-kube-api-access-knqwv" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122120    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c982442-0aef-45b4-acd0-16e707422ca9" volumeName="kubernetes.io/secret/9c982442-0aef-45b4-acd0-16e707422ca9-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122021    5224 factory.go:103] Registering Raw factory
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122131    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="086392f7-3abd-4c9b-ae7b-69ccd1985d1b" volumeName="kubernetes.io/secret/086392f7-3abd-4c9b-ae7b-69ccd1985d1b-agent-certs" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122143    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c982442-0aef-45b4-acd0-16e707422ca9" volumeName="kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-ovnkube-config" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122153    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af20298-29d1-41e2-8714-dbd4367aa89c" volumeName="kubernetes.io/configmap/3af20298-29d1-41e2-8714-dbd4367aa89c-iptables-alerter-script" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122155    5224 manager.go:1199] Started watching for new ooms in manager
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122164    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bd5801e-0707-4269-aef3-c22008c52025" volumeName="kubernetes.io/projected/3bd5801e-0707-4269-aef3-c22008c52025-kube-api-access-pqgql" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122177    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9201398f-e237-4997-9091-ffe08b8291ad" volumeName="kubernetes.io/empty-dir/9201398f-e237-4997-9091-ffe08b8291ad-etc-tuned" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122188    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9da3a466-b51b-46e5-850a-f40c9737a500" volumeName="kubernetes.io/projected/9da3a466-b51b-46e5-850a-f40c9737a500-kube-api-access-55smn" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122200    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="477caeb6-d7be-439d-a2e9-ce396eadb00b" volumeName="kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-config" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122211    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="539f14e3-83e4-4288-af2e-6dbd929a1565" volumeName="kubernetes.io/projected/539f14e3-83e4-4288-af2e-6dbd929a1565-kube-api-access-ls4kl" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122228    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0979491-9a8b-4ec4-b3e0-59ad16c4f454" volumeName="kubernetes.io/empty-dir/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-data-volume" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122254    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="086392f7-3abd-4c9b-ae7b-69ccd1985d1b" volumeName="kubernetes.io/configmap/086392f7-3abd-4c9b-ae7b-69ccd1985d1b-konnectivity-ca" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122267    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bd5801e-0707-4269-aef3-c22008c52025" volumeName="kubernetes.io/configmap/3bd5801e-0707-4269-aef3-c22008c52025-cni-binary-copy" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122280    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9201398f-e237-4997-9091-ffe08b8291ad" volumeName="kubernetes.io/empty-dir/9201398f-e237-4997-9091-ffe08b8291ad-tmp" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122296    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb5b4895-f27a-4de9-a084-4b6bf0a4af9d" volumeName="kubernetes.io/configmap/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122310    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb5b4895-f27a-4de9-a084-4b6bf0a4af9d" volumeName="kubernetes.io/secret/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-tls" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122322    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" volumeName="kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cni-binary-copy" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122341    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6a65e50-7411-4a9f-af62-92340babdbbd" volumeName="kubernetes.io/empty-dir/e6a65e50-7411-4a9f-af62-92340babdbbd-tmp" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122360    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ab2031f-ddc9-4e69-a860-4d326697de18" volumeName="kubernetes.io/secret/0ab2031f-ddc9-4e69-a860-4d326697de18-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122377    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="143007dd-d254-458e-b203-ba223e8ff61c" volumeName="kubernetes.io/secret/143007dd-d254-458e-b203-ba223e8ff61c-metrics-tls" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122389    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9da3a466-b51b-46e5-850a-f40c9737a500" volumeName="kubernetes.io/secret/9da3a466-b51b-46e5-850a-f40c9737a500-cert" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122409    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f" volumeName="kubernetes.io/empty-dir/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-tmp-dir" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122423    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" volumeName="kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122438    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="143007dd-d254-458e-b203-ba223e8ff61c" volumeName="kubernetes.io/projected/143007dd-d254-458e-b203-ba223e8ff61c-kube-api-access-mgwns" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122452    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="477caeb6-d7be-439d-a2e9-ce396eadb00b" volumeName="kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-service-ca" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122465    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="477caeb6-d7be-439d-a2e9-ce396eadb00b" volumeName="kubernetes.io/empty-dir/477caeb6-d7be-439d-a2e9-ce396eadb00b-tmp" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122478    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="477caeb6-d7be-439d-a2e9-ce396eadb00b" volumeName="kubernetes.io/projected/477caeb6-d7be-439d-a2e9-ce396eadb00b-kube-api-access-fp4ls" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122492    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0979491-9a8b-4ec4-b3e0-59ad16c4f454" volumeName="kubernetes.io/secret/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 02 19:48:39.122797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122505    5224 manager.go:319] Starting recovery of all containers
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.122505    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="256ea3f6-c170-4734-a39d-63fcc9049b83" volumeName="kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv" seLinuxMountContext=""
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.125140    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="857aedf0-97a2-469a-b0fd-765845566e58" volumeName="kubernetes.io/projected/857aedf0-97a2-469a-b0fd-765845566e58-kube-api-access-89wdg" seLinuxMountContext=""
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.125164    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c982442-0aef-45b4-acd0-16e707422ca9" volumeName="kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-env-overrides" seLinuxMountContext=""
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.125177    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="143007dd-d254-458e-b203-ba223e8ff61c" volumeName="kubernetes.io/empty-dir/143007dd-d254-458e-b203-ba223e8ff61c-tmp-dir" seLinuxMountContext=""
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.125190    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="220ffe56-58a3-4bd9-b93a-dcb1ac38fc89" volumeName="kubernetes.io/configmap/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-serviceca" seLinuxMountContext=""
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.125213    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="477caeb6-d7be-439d-a2e9-ce396eadb00b" volumeName="kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-oauth-config" seLinuxMountContext=""
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.125224    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9201398f-e237-4997-9091-ffe08b8291ad" volumeName="kubernetes.io/projected/9201398f-e237-4997-9091-ffe08b8291ad-kube-api-access-dhd5m" seLinuxMountContext=""
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.125243    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb5b4895-f27a-4de9-a084-4b6bf0a4af9d" volumeName="kubernetes.io/secret/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.125255    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0979491-9a8b-4ec4-b3e0-59ad16c4f454" volumeName="kubernetes.io/projected/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-kube-api-access-nvw5f" seLinuxMountContext=""
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.125267    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c982442-0aef-45b4-acd0-16e707422ca9" volumeName="kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.125337    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" volumeName="kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.125350    5224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4c78d9b-2b7d-4522-af17-f595eae2ab56" volumeName="kubernetes.io/secret/e4c78d9b-2b7d-4522-af17-f595eae2ab56-tls-certificates" seLinuxMountContext=""
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.125361    5224 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 19:48:39.129128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.125376    5224 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 19:48:39.158350 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.158283    5224 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 19:48:39.159875 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.159851    5224 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 19:48:39.159875 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.159879    5224 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 19:48:39.160039 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.159901    5224 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 19:48:39.160039 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:39.159942    5224 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 19:48:39.182028 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.181951    5224 generic.go:356] "Generic (PLEG): container finished" podID="f28b16d2fb7687a2bf2bca5db4901e34" containerID="6a595a2bb7dbd824256d77027eed46ebad0940ee2aa815380e14ac544e10577b" exitCode=0
Nov 02 19:48:39.198641 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.198612    5224 generic.go:356] "Generic (PLEG): container finished" podID="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" containerID="0ecd8bf629a8b1597ca45783c0ca33e0cdb71ca673d82b654a46bf7786ba41d9" exitCode=0
Nov 02 19:48:39.198641 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.198636    5224 generic.go:356] "Generic (PLEG): container finished" podID="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" containerID="7de362dcd97dd79b42cdfdaf818217c7eb9998abf6d9679b8c2e47135f217019" exitCode=0
Nov 02 19:48:39.198641 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.198643    5224 generic.go:356] "Generic (PLEG): container finished" podID="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" containerID="d4085aebd423166d7c9e333539e1b410830f60e0606cc3a552eef30640126bd5" exitCode=0
Nov 02 19:48:39.198641 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.198649    5224 generic.go:356] "Generic (PLEG): container finished" podID="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" containerID="1dee6805dc6a3e74de21dcc51d657a6e352077001e95e8805d56c98719fe569a" exitCode=0
Nov 02 19:48:39.198962 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.198655    5224 generic.go:356] "Generic (PLEG): container finished" podID="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" containerID="c2eabe2e7341b000bb8f03480e0ec6d3432e2c513f7eb9cba348c2ab9788e9d3" exitCode=0
Nov 02 19:48:39.198962 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.198660    5224 generic.go:356] "Generic (PLEG): container finished" podID="ce51ba2f-b8b5-4b53-8a8e-be70bda4df63" containerID="987d8f6f5d272f60c9da42195a68fdf932bebe89289503a81606f41342785026" exitCode=0
Nov 02 19:48:39.207235 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.207202    5224 generic.go:356] "Generic (PLEG): container finished" podID="cb5b4895-f27a-4de9-a084-4b6bf0a4af9d" containerID="70702ce38e7a17eccf2cb521663415784e3332ef39981c0ccb9eaa54db5f6a46" exitCode=0
Nov 02 19:48:39.220885 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:39.220850    5224 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-196.us-west-2.compute.internal\" not found"
Nov 02 19:48:39.224651 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.224630    5224 manager.go:324] Recovery completed
Nov 02 19:48:39.247162 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.247134    5224 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:48:39.252154 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.252135    5224 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:48:39.252263 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.252165    5224 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:48:39.252263 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.252180    5224 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:48:39.254488 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.254467    5224 cpu_manager.go:226] "Starting" policy="none"
Nov 02 19:48:39.254488 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.254482    5224 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 19:48:39.254644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.254495    5224 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:48:39.254644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.254500    5224 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 19:48:39.254741 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.254699    5224 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 19:48:39.254741 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.254713    5224 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 19:48:39.254741 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.254734    5224 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 19:48:39.254741 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.254742    5224 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 19:48:39.254908 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.254748    5224 policy_none.go:50] "Start"
Nov 02 19:48:39.254908 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.254759    5224 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 19:48:39.254908 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.254790    5224 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 19:48:39.255025 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.254954    5224 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 19:48:39.255025 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.254963    5224 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 19:48:39.255025 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.255000    5224 policy_none.go:44] "Start"
Nov 02 19:48:39.255516 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.255493    5224 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:48:39.255607 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.255527    5224 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:48:39.255607 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.255542    5224 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:48:39.260138 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:39.260118    5224 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 19:48:39.260230 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.260193    5224 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 19:48:39.263196 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.263182    5224 manager.go:341] "Starting Device Plugin manager"
Nov 02 19:48:39.263279 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:39.263211    5224 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 19:48:39.263279 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.263221    5224 server.go:87] "Starting device plugin registration server"
Nov 02 19:48:39.264063 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.264048    5224 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 19:48:39.264144 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.264064    5224 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 19:48:39.264223 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.264210    5224 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 19:48:39.264325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.264300    5224 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 19:48:39.264375 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.264335    5224 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 19:48:39.264375 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.264342    5224 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 19:48:39.264515 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.264400    5224 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:48:39.264323556Z","UUID":"d964024d-d7e7-4f4d-8ce8-2da7c067cfac","Handler":null,"Name":"","Endpoint":""}
Nov 02 19:48:39.266067 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.266047    5224 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 19:48:39.266067 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.266077    5224 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 19:48:39.266370 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:39.266344    5224 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 19:48:39.267066 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:39.267035    5224 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-196.us-west-2.compute.internal\" not found"
Nov 02 19:48:39.308730 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.308688    5224 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:48:39.308861 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.308787    5224 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:48:39.308861 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.308849    5224 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 19:48:39.310102 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.310078    5224 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:48:39.364232 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.364205    5224 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:48:39.365315 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.365290    5224 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:48:39.365419 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.365325    5224 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:48:39.365419 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.365340    5224 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-196.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:48:39.365419 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.365379    5224 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.410411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.410377    5224 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:48:39.461137 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.461037    5224 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"]
Nov 02 19:48:39.461710 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.461664    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal" event={"ID":"f28b16d2fb7687a2bf2bca5db4901e34","Type":"ContainerStarted","Data":"295cc0d09b51ef64c9d7cfac865b56c288339a16dfcf92002ffa93bbed7f28f1"}
Nov 02 19:48:39.461710 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.461719    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal" event={"ID":"f28b16d2fb7687a2bf2bca5db4901e34","Type":"ContainerDied","Data":"6a595a2bb7dbd824256d77027eed46ebad0940ee2aa815380e14ac544e10577b"}
Nov 02 19:48:39.461910 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.461742    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal" event={"ID":"f28b16d2fb7687a2bf2bca5db4901e34","Type":"ContainerStarted","Data":"6ca0a98c88ac99c89ccd69f236d073ecfeccf28369fa4e589b6887d032d308ae"}
Nov 02 19:48:39.461910 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.461820    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal" event={"ID":"09b897c9011dced250aef3fd51f6bd3f","Type":"ContainerStarted","Data":"0a9ae2fe32cb95823a3372f83628fe3d87f63e74a04b6f6ab9363a92849dbf51"}
Nov 02 19:48:39.461910 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.461839    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal" event={"ID":"09b897c9011dced250aef3fd51f6bd3f","Type":"ContainerStarted","Data":"f43d752c2ae6f2bf739bf9ce3c576f5bfdf1445401edd1a5576a254289b579e4"}
Nov 02 19:48:39.461910 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.461856    5224 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.461910 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.461871    5224 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.514338 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.514302    5224 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.514477 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.514380    5224 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.524140 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.524111    5224 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:48:39.524269 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:39.524173    5224 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.527108 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.527081    5224 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:48:39.527245 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:39.527121    5224 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.611380 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.611344    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09b897c9011dced250aef3fd51f6bd3f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"09b897c9011dced250aef3fd51f6bd3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.611544 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.611390    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f28b16d2fb7687a2bf2bca5db4901e34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"f28b16d2fb7687a2bf2bca5db4901e34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.611544 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.611411    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f28b16d2fb7687a2bf2bca5db4901e34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"f28b16d2fb7687a2bf2bca5db4901e34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.711605 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.711515    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09b897c9011dced250aef3fd51f6bd3f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"09b897c9011dced250aef3fd51f6bd3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.711605 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.711566    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f28b16d2fb7687a2bf2bca5db4901e34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"f28b16d2fb7687a2bf2bca5db4901e34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.711605 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.711596    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f28b16d2fb7687a2bf2bca5db4901e34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"f28b16d2fb7687a2bf2bca5db4901e34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.711857 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.711626    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09b897c9011dced250aef3fd51f6bd3f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"09b897c9011dced250aef3fd51f6bd3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.711857 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.711647    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f28b16d2fb7687a2bf2bca5db4901e34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"f28b16d2fb7687a2bf2bca5db4901e34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:39.711857 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:39.711709    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f28b16d2fb7687a2bf2bca5db4901e34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" (UID: \"f28b16d2fb7687a2bf2bca5db4901e34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:40.094803 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.094762    5224 apiserver.go:52] "Watching apiserver"
Nov 02 19:48:40.232247 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.232220    5224 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:40.232438 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.232407    5224 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:40.255966 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.255942    5224 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 19:48:40.256754 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.256727    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n9zfw","openshift-network-diagnostics/network-check-target-hxstf","openshift-dns/node-resolver-h6nfx","openshift-ingress-canary/ingress-canary-clwhz","openshift-network-operator/iptables-alerter-k46lq","openshift-console/downloads-6dd789c8b7-cchgk","openshift-dns/dns-default-pgp55","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb","openshift-multus/multus-additional-cni-plugins-h6b6p","openshift-multus/network-metrics-daemon-qcr2p","kube-system/global-pull-secret-syncer-kf87v","kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal","openshift-monitoring/alertmanager-main-0","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k","openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd","openshift-console/console-794cd4c8c5-td84z","openshift-insights/insights-runtime-extractor-6kgdb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-w4m6t","openshift-multus/multus-rsl26","kube-system/konnectivity-agent-jwvp7","openshift-cluster-node-tuning-operator/tuned-mm2bm","openshift-image-registry/node-ca-qxbct"]
Nov 02 19:48:40.272083 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.272055    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.295448 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.295422    5224 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:48:40.295589 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:40.295481    5224 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:40.295589 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.295482    5224 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:48:40.295681 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:40.295604    5224 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-196.us-west-2.compute.internal"
Nov 02 19:48:40.317275 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.317243    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 19:48:40.317399 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.317333    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 19:48:40.317444 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.317431    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 19:48:40.317517 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.317500    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 19:48:40.317639 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.317619    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 19:48:40.317855 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.317833    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 19:48:40.317975 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.317908    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 19:48:40.318030 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.317986    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 19:48:40.318079 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.318069    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 19:48:40.318147 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.318134    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 19:48:40.318247 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.318233    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 19:48:40.318315 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.318302    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 19:48:40.318393 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.318380    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 19:48:40.318492 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.318478    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 19:48:40.318566 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.318553    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:48:40.318639 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.318629    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 19:48:40.318715 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.318703    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 19:48:40.318841 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.318827    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 19:48:40.321043 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.321025    5224 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 19:48:40.401417 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.401336    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 19:48:40.401662 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.401637    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7cqmh\""
Nov 02 19:48:40.401957 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.401922    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 19:48:40.402259 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402235    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 19:48:40.402362 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402295    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 19:48:40.402362 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402321    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kkdt\""
Nov 02 19:48:40.402362 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402352    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7zhgs\""
Nov 02 19:48:40.402511 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402490    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 19:48:40.402607 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402495    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 19:48:40.402607 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402580    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fhf7l\""
Nov 02 19:48:40.402607 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402593    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ww82t\""
Nov 02 19:48:40.402791 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402618    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6t9zq\""
Nov 02 19:48:40.402791 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402634    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 19:48:40.402791 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402687    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 19:48:40.402791 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402691    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x9wbn\""
Nov 02 19:48:40.402791 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402721    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 02 19:48:40.402791 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402723    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 19:48:40.402791 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402779    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 19:48:40.403090 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402810    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 19:48:40.403090 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402819    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 19:48:40.403090 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402825    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 19:48:40.403090 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402843    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 19:48:40.403090 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402855    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-x9fws\""
Nov 02 19:48:40.403090 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402899    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 19:48:40.403090 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.402911    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-z7kgr\""
Nov 02 19:48:40.403090 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403008    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7q2qk\""
Nov 02 19:48:40.403090 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403089    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 19:48:40.403502 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403124    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bv4cs\""
Nov 02 19:48:40.403502 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403194    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-cjnzq\""
Nov 02 19:48:40.403502 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403206    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 02 19:48:40.403502 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403230    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-t7wfl\""
Nov 02 19:48:40.403502 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403387    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wsdm2\""
Nov 02 19:48:40.403502 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403398    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rmffj\""
Nov 02 19:48:40.403502 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403408    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qzg2s\""
Nov 02 19:48:40.403502 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403408    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tnfsn\""
Nov 02 19:48:40.403502 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403432    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p97wz\""
Nov 02 19:48:40.403502 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403471    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 19:48:40.403502 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403481    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xk7lc\""
Nov 02 19:48:40.403949 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403554    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dsl5q\""
Nov 02 19:48:40.403949 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403611    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 19:48:40.403949 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403686    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 19:48:40.403949 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403744    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 19:48:40.403949 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403839    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 19:48:40.403949 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.403917    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v8hmw\""
Nov 02 19:48:40.414595 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.414574    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:48:40.414595 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.414579    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 19:48:40.414945 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.414918    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 19:48:40.415044 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.414974    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 19:48:40.415120 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.415106    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 19:48:40.415120 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.415115    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 19:48:40.415215 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.415201    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 19:48:40.415537 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.415506    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 19:48:40.419901 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.419876    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89wdg\" (UniqueName: \"kubernetes.io/projected/857aedf0-97a2-469a-b0fd-765845566e58-kube-api-access-89wdg\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.419994 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.419917    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-out\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.419994 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.419953    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/143007dd-d254-458e-b203-ba223e8ff61c-config-volume\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:40.419994 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.419973    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-crio-socket\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:40.420106 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420046    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls4kl\" (UniqueName: \"kubernetes.io/projected/539f14e3-83e4-4288-af2e-6dbd929a1565-kube-api-access-ls4kl\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:40.420154 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420117    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-989mm\" (UniqueName: \"kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-kube-api-access-989mm\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.420276 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420258    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cni-binary-copy\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.420367 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420336    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-sys\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.420449 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420434    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.420487 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420441    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/143007dd-d254-458e-b203-ba223e8ff61c-config-volume\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:40.420547 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420497    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-lib-modules\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.420547 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420532    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cni-binary-copy\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.420653 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420574    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhd5m\" (UniqueName: \"kubernetes.io/projected/9201398f-e237-4997-9091-ffe08b8291ad-kube-api-access-dhd5m\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.420653 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420638    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af20298-29d1-41e2-8714-dbd4367aa89c-iptables-alerter-script\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:48:40.420753 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420731    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-run-netns\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.420861 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420798    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-env-overrides\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.420925 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420899    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-tmp-dir\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:48:40.420975 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420938    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af20298-29d1-41e2-8714-dbd4367aa89c-iptables-alerter-script\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:48:40.421029 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420965    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-env-overrides\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.421029 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420993    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsss9\" (UniqueName: \"kubernetes.io/projected/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-kube-api-access-zsss9\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:48:40.421131 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421024    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-socket-dir-parent\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.421131 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421058    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-etc-kubernetes\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.421131 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.420983    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-tmp-dir\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:48:40.421131 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421081    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysctl-conf\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.421131 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421110    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.421359 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421136    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-service-ca\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.421359 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421235    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-hosts-file\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:48:40.421359 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421290    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.421359 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421321    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:40.421359 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421354    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-textfile\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.421580 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421363    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-service-ca\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.421580 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421378    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-wtmp\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.421580 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421420    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.421580 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421461    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-textfile\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.421580 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421476    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ab2031f-ddc9-4e69-a860-4d326697de18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:40.421580 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421518    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg7nf\" (UniqueName: \"kubernetes.io/projected/e6a65e50-7411-4a9f-af62-92340babdbbd-kube-api-access-fg7nf\") pod \"downloads-6dd789c8b7-cchgk\" (UID: \"e6a65e50-7411-4a9f-af62-92340babdbbd\") " pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:48:40.421863 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421698    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-kubelet\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.421863 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421725    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqwv\" (UniqueName: \"kubernetes.io/projected/9c982442-0aef-45b4-acd0-16e707422ca9-kube-api-access-knqwv\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.421863 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421753    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ab2031f-ddc9-4e69-a860-4d326697de18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:40.421863 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421778    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-netns\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.421863 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421813    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqgql\" (UniqueName: \"kubernetes.io/projected/3bd5801e-0707-4269-aef3-c22008c52025-kube-api-access-pqgql\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.421863 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421846    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.422074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421958    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-etc-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.422074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.421997    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-ovnkube-config\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.422074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422013    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-conf-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.422074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422026    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysctl-d\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.422074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422044    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab2031f-ddc9-4e69-a860-4d326697de18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:40.422074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422071    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.422314 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422121    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-ovn\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.422314 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422137    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.422314 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422152    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-multus-certs\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.422314 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422174    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-tls\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.422314 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422240    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-ovnkube-config\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.422314 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422296    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-systemd-units\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.422552 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422321    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-slash\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.422552 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422341    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-k8s-cni-cncf-io\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.422552 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422368    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-var-lib-kubelet\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.422552 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422392    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-tls\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.422552 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422416    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-host\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.422552 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422439    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5fqq\" (UniqueName: \"kubernetes.io/projected/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-kube-api-access-r5fqq\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.422552 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422464    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd0732dd-9b73-4204-a64b-ba95c2aa6007-kubelet-config\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:40.422552 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422484    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:40.422552 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422523    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-systemd\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.422871 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422561    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-root\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.422871 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422577    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.422871 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422595    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.422871 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422662    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.422871 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422696    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-oauth-config\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.422871 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422723    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.422871 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422752    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cnibin\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.422871 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422806    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd0732dd-9b73-4204-a64b-ba95c2aa6007-dbus\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:40.422871 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422833    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:40.423133 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422923    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-system-cni-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.423133 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422949    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9201398f-e237-4997-9091-ffe08b8291ad-tmp\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.423133 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.422980    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-sys\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.423133 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423004    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/477caeb6-d7be-439d-a2e9-ce396eadb00b-tmp\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.423133 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423043    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/086392f7-3abd-4c9b-ae7b-69ccd1985d1b-agent-certs\") pod \"konnectivity-agent-jwvp7\" (UID: \"086392f7-3abd-4c9b-ae7b-69ccd1985d1b\") " pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:48:40.423133 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423066    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-var-lib-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.423133 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423089    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-cni-bin\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.423133 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423117    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/086392f7-3abd-4c9b-ae7b-69ccd1985d1b-konnectivity-ca\") pod \"konnectivity-agent-jwvp7\" (UID: \"086392f7-3abd-4c9b-ae7b-69ccd1985d1b\") " pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:48:40.423388 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423143    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab2031f-ddc9-4e69-a860-4d326697de18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:40.423388 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423208    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-config\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.423388 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423234    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.423388 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423262    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.423545 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423448    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-kubernetes\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.423545 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423478    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-serviceca\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:48:40.423545 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423511    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-socket-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.423639 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423542    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-systemd\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.423639 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423567    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3bd5801e-0707-4269-aef3-c22008c52025-multus-daemon-config\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.423639 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423601    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:40.423639 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423631    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.423792 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423656    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.423792 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423682    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pj2b\" (UniqueName: \"kubernetes.io/projected/3af20298-29d1-41e2-8714-dbd4367aa89c-kube-api-access-4pj2b\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:48:40.423792 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423706    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.423792 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423732    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwns\" (UniqueName: \"kubernetes.io/projected/143007dd-d254-458e-b203-ba223e8ff61c-kube-api-access-mgwns\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:40.423792 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423786    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-cni-multus\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.424021 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423822    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-kubelet\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.424021 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423852    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp4ls\" (UniqueName: \"kubernetes.io/projected/477caeb6-d7be-439d-a2e9-ce396eadb00b-kube-api-access-fp4ls\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.424021 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423878    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.424021 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423901    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.424021 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423924    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-cni-netd\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.424021 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423947    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-node-log\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.424021 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423970    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-log-socket\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.424021 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.423996    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/143007dd-d254-458e-b203-ba223e8ff61c-metrics-tls\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:40.424385 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424026    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmbv\" (UniqueName: \"kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv\") pod \"network-check-target-hxstf\" (UID: \"256ea3f6-c170-4734-a39d-63fcc9049b83\") " pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:40.424385 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424053    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bd5801e-0707-4269-aef3-c22008c52025-cni-binary-copy\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.424385 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424085    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-hostroot\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.424385 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424123    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.424385 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424148    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 19:48:40.424385 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424209    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 19:48:40.424385 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424307    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-run\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.424385 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424337    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-registration-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.424709 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424668    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5drg\" (UniqueName: \"kubernetes.io/projected/0ab2031f-ddc9-4e69-a860-4d326697de18-kube-api-access-n5drg\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:40.424785 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424745    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-cni-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.424834 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424809    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvw5f\" (UniqueName: \"kubernetes.io/projected/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-kube-api-access-nvw5f\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:40.424878 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424847    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e4c78d9b-2b7d-4522-af17-f595eae2ab56-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lbcmb\" (UID: \"e4c78d9b-2b7d-4522-af17-f595eae2ab56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb"
Nov 02 19:48:40.424934 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424879    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-volume\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.424934 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424913    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55smn\" (UniqueName: \"kubernetes.io/projected/9da3a466-b51b-46e5-850a-f40c9737a500-kube-api-access-55smn\") pod \"ingress-canary-clwhz\" (UID: \"9da3a466-b51b-46e5-850a-f40c9737a500\") " pod="openshift-ingress-canary/ingress-canary-clwhz"
Nov 02 19:48:40.425022 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424939    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6a65e50-7411-4a9f-af62-92340babdbbd-tmp\") pod \"downloads-6dd789c8b7-cchgk\" (UID: \"e6a65e50-7411-4a9f-af62-92340babdbbd\") " pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:48:40.425022 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424968    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 19:48:40.425137 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425118    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 19:48:40.425359 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425306    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 19:48:40.425447 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.424973    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af20298-29d1-41e2-8714-dbd4367aa89c-host-slash\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:48:40.425499 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425471    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-host\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:48:40.425499 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425495    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-system-cni-dir\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.425555 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425520    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143007dd-d254-458e-b203-ba223e8ff61c-tmp-dir\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:40.425555 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425527    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e4c78d9b-2b7d-4522-af17-f595eae2ab56-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lbcmb\" (UID: \"e4c78d9b-2b7d-4522-af17-f595eae2ab56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb"
Nov 02 19:48:40.425555 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425544    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-cni-bin\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.425639 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425565    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-modprobe-d\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.425639 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425600    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmxqn\" (UniqueName: \"kubernetes.io/projected/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-kube-api-access-gmxqn\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:48:40.425709 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425676    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-serving-cert\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.425709 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425700    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-ovnkube-script-lib\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.425786 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425718    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-cnibin\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.425786 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425738    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysconfig\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.425786 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425759    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9201398f-e237-4997-9091-ffe08b8291ad-etc-tuned\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425932    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-metrics-client-ca\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425947    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9201398f-e237-4997-9091-ffe08b8291ad-etc-tuned\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.425979    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-sys-fs\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426011    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143007dd-d254-458e-b203-ba223e8ff61c-tmp-dir\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426052    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6a65e50-7411-4a9f-af62-92340babdbbd-tmp\") pod \"downloads-6dd789c8b7-cchgk\" (UID: \"e6a65e50-7411-4a9f-af62-92340babdbbd\") " pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426201    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9da3a466-b51b-46e5-850a-f40c9737a500-cert\") pod \"ingress-canary-clwhz\" (UID: \"9da3a466-b51b-46e5-850a-f40c9737a500\") " pod="openshift-ingress-canary/ingress-canary-clwhz"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426256    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-data-volume\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426308    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c982442-0aef-45b4-acd0-16e707422ca9-ovn-node-metrics-cert\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426319    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426329    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539f14e3-83e4-4288-af2e-6dbd929a1565-metrics-certs\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426353    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62972\" (UniqueName: \"kubernetes.io/projected/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-kube-api-access-62972\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426461    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9da3a466-b51b-46e5-850a-f40c9737a500-cert\") pod \"ingress-canary-clwhz\" (UID: \"9da3a466-b51b-46e5-850a-f40c9737a500\") " pod="openshift-ingress-canary/ingress-canary-clwhz"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426552    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0732dd-9b73-4204-a64b-ba95c2aa6007-original-pull-secret\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426709    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9201398f-e237-4997-9091-ffe08b8291ad-tmp\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426815    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426973    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c982442-0aef-45b4-acd0-16e707422ca9-ovn-node-metrics-cert\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.426997    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-config\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427064    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-data-volume\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427079    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/477caeb6-d7be-439d-a2e9-ce396eadb00b-tmp\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427215    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bd5801e-0707-4269-aef3-c22008c52025-cni-binary-copy\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427265    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427324    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-oauth-serving-cert\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427553    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-serviceca\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427574    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-oauth-config\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427574    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-oauth-serving-cert\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427642    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/143007dd-d254-458e-b203-ba223e8ff61c-metrics-tls\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427679    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-web-config\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427718    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-device-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427801    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c982442-0aef-45b4-acd0-16e707422ca9-ovnkube-script-lib\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427908    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-serving-cert\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.427967    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.428097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.428013    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:40.429486 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.428124    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab2031f-ddc9-4e69-a860-4d326697de18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:40.429486 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.428251    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/086392f7-3abd-4c9b-ae7b-69ccd1985d1b-konnectivity-ca\") pod \"konnectivity-agent-jwvp7\" (UID: \"086392f7-3abd-4c9b-ae7b-69ccd1985d1b\") " pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:48:40.429486 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.428313    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.429486 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.428328    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/086392f7-3abd-4c9b-ae7b-69ccd1985d1b-agent-certs\") pod \"konnectivity-agent-jwvp7\" (UID: \"086392f7-3abd-4c9b-ae7b-69ccd1985d1b\") " pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:48:40.432140 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.432119    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:40.452260 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.452236    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 19:48:40.452340 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.452244    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 19:48:40.452695 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.452653    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 19:48:40.452695 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.452666    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 19:48:40.452695 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.452688    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 19:48:40.453183 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.452792    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 19:48:40.453183 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.452828    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 19:48:40.453183 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.452853    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 19:48:40.453555 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.453537    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 19:48:40.457717 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.457696    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.457800 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.457720    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-metrics-client-ca\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.458887 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.458868    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3bd5801e-0707-4269-aef3-c22008c52025-multus-daemon-config\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.463048 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.463032    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab2031f-ddc9-4e69-a860-4d326697de18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:40.477098 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.477068    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89wdg\" (UniqueName: \"kubernetes.io/projected/857aedf0-97a2-469a-b0fd-765845566e58-kube-api-access-89wdg\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.477305 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.477283    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhd5m\" (UniqueName: \"kubernetes.io/projected/9201398f-e237-4997-9091-ffe08b8291ad-kube-api-access-dhd5m\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.477398 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.477305    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsss9\" (UniqueName: \"kubernetes.io/projected/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-kube-api-access-zsss9\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:48:40.477463 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.477446    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls4kl\" (UniqueName: \"kubernetes.io/projected/539f14e3-83e4-4288-af2e-6dbd929a1565-kube-api-access-ls4kl\") pod \"network-metrics-daemon-qcr2p\" (UID: \"539f14e3-83e4-4288-af2e-6dbd929a1565\") " pod="openshift-multus/network-metrics-daemon-qcr2p"
Nov 02 19:48:40.477821 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.477802    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqwv\" (UniqueName: \"kubernetes.io/projected/9c982442-0aef-45b4-acd0-16e707422ca9-kube-api-access-knqwv\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.478561 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.478541    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqgql\" (UniqueName: \"kubernetes.io/projected/3bd5801e-0707-4269-aef3-c22008c52025-kube-api-access-pqgql\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.478777 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.478747    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg7nf\" (UniqueName: \"kubernetes.io/projected/e6a65e50-7411-4a9f-af62-92340babdbbd-kube-api-access-fg7nf\") pod \"downloads-6dd789c8b7-cchgk\" (UID: \"e6a65e50-7411-4a9f-af62-92340babdbbd\") " pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:48:40.480935 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.480911    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvw5f\" (UniqueName: \"kubernetes.io/projected/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-kube-api-access-nvw5f\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:40.482287 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.482259    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55smn\" (UniqueName: \"kubernetes.io/projected/9da3a466-b51b-46e5-850a-f40c9737a500-kube-api-access-55smn\") pod \"ingress-canary-clwhz\" (UID: \"9da3a466-b51b-46e5-850a-f40c9737a500\") " pod="openshift-ingress-canary/ingress-canary-clwhz"
Nov 02 19:48:40.483349 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.483329    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmxqn\" (UniqueName: \"kubernetes.io/projected/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-kube-api-access-gmxqn\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:48:40.483601 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.483579    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp4ls\" (UniqueName: \"kubernetes.io/projected/477caeb6-d7be-439d-a2e9-ce396eadb00b-kube-api-access-fp4ls\") pod \"console-794cd4c8c5-td84z\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") " pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:40.484345 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.484325    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pj2b\" (UniqueName: \"kubernetes.io/projected/3af20298-29d1-41e2-8714-dbd4367aa89c-kube-api-access-4pj2b\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:48:40.484698 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.484682    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cmbv\" (UniqueName: \"kubernetes.io/projected/256ea3f6-c170-4734-a39d-63fcc9049b83-kube-api-access-7cmbv\") pod \"network-check-target-hxstf\" (UID: \"256ea3f6-c170-4734-a39d-63fcc9049b83\") " pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:40.484871 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.484855    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwns\" (UniqueName: \"kubernetes.io/projected/143007dd-d254-458e-b203-ba223e8ff61c-kube-api-access-mgwns\") pod \"dns-default-pgp55\" (UID: \"143007dd-d254-458e-b203-ba223e8ff61c\") " pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:40.485239 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.485220    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62972\" (UniqueName: \"kubernetes.io/projected/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-kube-api-access-62972\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.485343 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.485308    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5drg\" (UniqueName: \"kubernetes.io/projected/0ab2031f-ddc9-4e69-a860-4d326697de18-kube-api-access-n5drg\") pod \"prometheus-operator-9cd89b74c-mfdwd\" (UID: \"0ab2031f-ddc9-4e69-a860-4d326697de18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mfdwd"
Nov 02 19:48:40.485703 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.485688    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5fqq\" (UniqueName: \"kubernetes.io/projected/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-kube-api-access-r5fqq\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.528427 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528392    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-volume\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.528427 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528436    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af20298-29d1-41e2-8714-dbd4367aa89c-host-slash\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:48:40.528676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528461    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-host\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:48:40.528676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528491    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-system-cni-dir\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.528676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528507    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-cni-bin\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.528676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528524    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-modprobe-d\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.528676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528538    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af20298-29d1-41e2-8714-dbd4367aa89c-host-slash\") pod \"iptables-alerter-k46lq\" (UID: \"3af20298-29d1-41e2-8714-dbd4367aa89c\") " pod="openshift-network-operator/iptables-alerter-k46lq"
Nov 02 19:48:40.528676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528546    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-cnibin\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.528676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528606    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/220ffe56-58a3-4bd9-b93a-dcb1ac38fc89-host\") pod \"node-ca-qxbct\" (UID: \"220ffe56-58a3-4bd9-b93a-dcb1ac38fc89\") " pod="openshift-image-registry/node-ca-qxbct"
Nov 02 19:48:40.528676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528610    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-cnibin\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.528676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528609    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysconfig\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.528676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528655    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysconfig\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.528676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528656    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-sys-fs\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528697    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-sys-fs\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528707    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-web-config\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528717    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-modprobe-d\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528731    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-device-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528762    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-out\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528792    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-device-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528801    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-crio-socket\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528820    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-989mm\" (UniqueName: \"kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-kube-api-access-989mm\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528829    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-cni-bin\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528834    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-sys\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528858    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528888    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-lib-modules\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528907    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-run-netns\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528934    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-socket-dir-parent\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528957    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-etc-kubernetes\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528980    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysctl-conf\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529004    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.528574    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-system-cni-dir\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529021    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-hosts-file\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529044    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529063    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-wtmp\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529077    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529099    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-kubelet\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529122    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-netns\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529126    5224 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529146    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-etc-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529162    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-conf-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529176    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysctl-d\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529200    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529199    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysctl-conf\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.529207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529223    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-ovn\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529244    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-multus-certs\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529260    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-systemd-units\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529283    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-slash\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529298    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-lib-modules\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529324    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-k8s-cni-cncf-io\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529334    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-run-netns\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529348    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-var-lib-kubelet\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529376    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-socket-dir-parent\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529379    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-host\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529401    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-etc-kubernetes\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529404    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd0732dd-9b73-4204-a64b-ba95c2aa6007-kubelet-config\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529434    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0979491-9a8b-4ec4-b3e0-59ad16c4f454-crio-socket\") pod \"insights-runtime-extractor-6kgdb\" (UID: \"f0979491-9a8b-4ec4-b3e0-59ad16c4f454\") " pod="openshift-insights/insights-runtime-extractor-6kgdb"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529428    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-systemd\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529476    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-root\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529498    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529530    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529556    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529579    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cnibin\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529616    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd0732dd-9b73-4204-a64b-ba95c2aa6007-dbus\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529641    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-system-cni-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529664    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-sys\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529689    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-var-lib-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529711    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-cni-bin\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529744    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529789    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529835    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-kubernetes\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529861    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-socket-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529879    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529891    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-systemd\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529940    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529946    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-systemd\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529964    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-sys\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529971    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.529998    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-cni-multus\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530017    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-host\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530024    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-kubelet\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530019    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530065    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530074    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-root\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530077    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd0732dd-9b73-4204-a64b-ba95c2aa6007-kubelet-config\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530093    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530110    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530172    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-etc-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530176    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f-hosts-file\") pod \"node-resolver-h6nfx\" (UID: \"bfc7f8e6-1a74-463a-9d86-2fc565f3fe0f\") " pod="openshift-dns/node-resolver-h6nfx"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530209    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-conf-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530250    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-var-lib-kubelet\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530281    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-slash\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.530432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530389    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-sysctl-d\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530470    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530532    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-kubernetes\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530587    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-socket-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530623    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-kubelet\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530663    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-ovn\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530709    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-node-exporter-wtmp\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530744    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-cnibin\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530760    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-etc-systemd\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530793    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-cni-bin\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530799    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-cni-netd\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530811    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-run-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530835    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-cni-netd\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530860    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd0732dd-9b73-4204-a64b-ba95c2aa6007-dbus\") pod \"global-pull-secret-syncer-kf87v\" (UID: \"cd0732dd-9b73-4204-a64b-ba95c2aa6007\") " pod="kube-system/global-pull-secret-syncer-kf87v"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530866    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce51ba2f-b8b5-4b53-8a8e-be70bda4df63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h6b6p\" (UID: \"ce51ba2f-b8b5-4b53-8a8e-be70bda4df63\") " pod="openshift-multus/multus-additional-cni-plugins-h6b6p"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530888    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530893    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-var-lib-cni-multus\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530907    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-var-lib-openvswitch\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530937    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530948    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-host-kubelet\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530986    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-system-cni-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.530989    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-systemd-units\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531020    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-multus-certs\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531028    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-netns\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531061    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-host-run-k8s-cni-cncf-io\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531117    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb5b4895-f27a-4de9-a084-4b6bf0a4af9d-sys\") pod \"node-exporter-n9zfw\" (UID: \"cb5b4895-f27a-4de9-a084-4b6bf0a4af9d\") " pod="openshift-monitoring/node-exporter-n9zfw"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531353    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-node-log\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531384    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-log-socket\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531413    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-hostroot\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531438    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531461    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-run\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531486    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-registration-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531512    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-cni-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531529    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531598    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-hostroot\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531657    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bd5801e-0707-4269-aef3-c22008c52025-multus-cni-dir\") pod \"multus-rsl26\" (UID: \"3bd5801e-0707-4269-aef3-c22008c52025\") " pod="openshift-multus/multus-rsl26"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531692    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-node-log\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531725    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c982442-0aef-45b4-acd0-16e707422ca9-log-socket\") pod \"ovnkube-node-w4m6t\" (UID: \"9c982442-0aef-45b4-acd0-16e707422ca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531780    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9201398f-e237-4997-9091-ffe08b8291ad-run\") pod \"tuned-mm2bm\" (UID: \"9201398f-e237-4997-9091-ffe08b8291ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mm2bm"
Nov 02 19:48:40.532368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.531825    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/857aedf0-97a2-469a-b0fd-765845566e58-registration-dir\") pod \"aws-ebs-csi-driver-node-dgk6k\" (UID: \"857aedf0-97a2-469a-b0fd-765845566e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgk6k"
Nov 02 19:48:40.533953 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.533263    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-volume\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.534223 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.534178    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.535328 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.535306    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.535495 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.535479    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-out\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.535562 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.535496    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.535612 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.535577    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.535816 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.535756    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.536189 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.536170    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-web-config\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.537344 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.537324    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.569979 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.569958    5224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:48:40.588145 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.588113    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-989mm\" (UniqueName: \"kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-kube-api-access-989mm\") pod \"alertmanager-main-0\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.590788 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.590744    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:48:40.792520 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:40.792484    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pgp55"
Nov 02 19:48:41.058085 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:41.058047    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d78e17_113c_4787_8a1d_5ead8f950bfc.slice/crio-36dd88251280c84e26c5153af2cd5790542a9afd8b2a17fedddb4ee56ec2ab59 WatchSource:0}: Error finding container 36dd88251280c84e26c5153af2cd5790542a9afd8b2a17fedddb4ee56ec2ab59: Status 404 returned error can't find the container with id 36dd88251280c84e26c5153af2cd5790542a9afd8b2a17fedddb4ee56ec2ab59
Nov 02 19:48:41.059822 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:41.059794    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 19:48:41.241662 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:41.241617    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerStarted","Data":"36dd88251280c84e26c5153af2cd5790542a9afd8b2a17fedddb4ee56ec2ab59"}
Nov 02 19:48:42.249029 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:42.248439    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794cd4c8c5-td84z" event={"ID":"477caeb6-d7be-439d-a2e9-ce396eadb00b","Type":"ContainerStarted","Data":"11d9f9816cd7150e36554e7fa82b104b975c95e11169181e469ee4d7af096dc5"}
Nov 02 19:48:42.676473 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:42.676433    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:42.845284 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:42.845257    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:42.854365 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:42.854335    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:42.976856 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:42.976762    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"]
Nov 02 19:48:42.983233 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:42.983209    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.037274 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.037242    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 19:48:43.037563 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.037535    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 19:48:43.037690 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.037611    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 19:48:43.037788 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.037701    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 19:48:43.037788 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.037746    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 19:48:43.037968 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.037943    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zp5f7\""
Nov 02 19:48:43.050945 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.050922    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/92b5e340-4e43-45e1-85e3-a867fda1c9f6-secret-telemeter-client\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.051070 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.050966    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92b5e340-4e43-45e1-85e3-a867fda1c9f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.051070 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.051011    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92b5e340-4e43-45e1-85e3-a867fda1c9f6-serving-certs-ca-bundle\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.051070 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.051028    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/92b5e340-4e43-45e1-85e3-a867fda1c9f6-telemeter-client-tls\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.051240 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.051141    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92b5e340-4e43-45e1-85e3-a867fda1c9f6-metrics-client-ca\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.051273 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.051246    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8bwk\" (UniqueName: \"kubernetes.io/projected/92b5e340-4e43-45e1-85e3-a867fda1c9f6-kube-api-access-s8bwk\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.051313 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.051299    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/92b5e340-4e43-45e1-85e3-a867fda1c9f6-federate-client-tls\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.051346 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.051328    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92b5e340-4e43-45e1-85e3-a867fda1c9f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.089513 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.089486    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 19:48:43.152597 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.152560    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92b5e340-4e43-45e1-85e3-a867fda1c9f6-serving-certs-ca-bundle\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.152597 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.152607    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/92b5e340-4e43-45e1-85e3-a867fda1c9f6-telemeter-client-tls\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.152842 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.152634    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92b5e340-4e43-45e1-85e3-a867fda1c9f6-metrics-client-ca\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.152842 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.152674    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8bwk\" (UniqueName: \"kubernetes.io/projected/92b5e340-4e43-45e1-85e3-a867fda1c9f6-kube-api-access-s8bwk\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.152842 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.152811    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/92b5e340-4e43-45e1-85e3-a867fda1c9f6-federate-client-tls\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.153053 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.152854    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92b5e340-4e43-45e1-85e3-a867fda1c9f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.153053 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.152889    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/92b5e340-4e43-45e1-85e3-a867fda1c9f6-secret-telemeter-client\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.153053 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.152917    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92b5e340-4e43-45e1-85e3-a867fda1c9f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.154396 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.154368    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92b5e340-4e43-45e1-85e3-a867fda1c9f6-serving-certs-ca-bundle\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.157143 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.154830    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92b5e340-4e43-45e1-85e3-a867fda1c9f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.157143 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.155210    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92b5e340-4e43-45e1-85e3-a867fda1c9f6-metrics-client-ca\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.157698 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.157670    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/92b5e340-4e43-45e1-85e3-a867fda1c9f6-telemeter-client-tls\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.161872 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.161837    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/92b5e340-4e43-45e1-85e3-a867fda1c9f6-federate-client-tls\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.163266 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.163240    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/92b5e340-4e43-45e1-85e3-a867fda1c9f6-secret-telemeter-client\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.164967 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.164944    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92b5e340-4e43-45e1-85e3-a867fda1c9f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.181854 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.181832    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"]
Nov 02 19:48:43.211251 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.211221    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8bwk\" (UniqueName: \"kubernetes.io/projected/92b5e340-4e43-45e1-85e3-a867fda1c9f6-kube-api-access-s8bwk\") pod \"telemeter-client-5dcb85b9b6-7xp5g\" (UID: \"92b5e340-4e43-45e1-85e3-a867fda1c9f6\") " pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.253870 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.253798    5224 generic.go:356] "Generic (PLEG): container finished" podID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerID="e82bbc55b9369caa40791b5668890d12e14fae0651c19ef70ec21471a2c548e5" exitCode=0
Nov 02 19:48:43.254300 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.253880    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerDied","Data":"e82bbc55b9369caa40791b5668890d12e14fae0651c19ef70ec21471a2c548e5"}
Nov 02 19:48:43.304324 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.304287    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"
Nov 02 19:48:43.721581 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:43.721553    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b5e340_4e43_45e1_85e3_a867fda1c9f6.slice/crio-adc71782533004fbced785c5b8a6ba0f7f74a808be6bcae68b73d7f80ca3cf85 WatchSource:0}: Error finding container adc71782533004fbced785c5b8a6ba0f7f74a808be6bcae68b73d7f80ca3cf85: Status 404 returned error can't find the container with id adc71782533004fbced785c5b8a6ba0f7f74a808be6bcae68b73d7f80ca3cf85
Nov 02 19:48:43.723112 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.723086    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g"]
Nov 02 19:48:43.905759 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:43.905726    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:48:44.259978 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:44.259936    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g" event={"ID":"92b5e340-4e43-45e1-85e3-a867fda1c9f6","Type":"ContainerStarted","Data":"adc71782533004fbced785c5b8a6ba0f7f74a808be6bcae68b73d7f80ca3cf85"}
Nov 02 19:48:44.645207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:44.645172    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:44.842421 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:44.842381    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:45.301782 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:45.301726    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:45.475637 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:45.475593    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:45.854062 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:45.854010    5224 log.go:32] "PullImage from image service failed" err="rpc error: code = Canceled desc = unable to pull image: copying system image from manifest list: copying config: context canceled" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36c3da5821b9e51334e9c523aca7fca6a9ef8b853fd4f238b908b086b129619d"
Nov 02 19:48:45.854362 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:45.854272    5224 kuberuntime_manager.go:1707] "Unhandled Error" err="container download-server start failed in pod downloads-6dd789c8b7-cchgk_openshift-console(e6a65e50-7411-4a9f-af62-92340babdbbd): ErrImagePull: rpc error: code = Canceled desc = unable to pull image: copying system image from manifest list: copying config: context canceled" logger="UnhandledError"
Nov 02 19:48:45.854362 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:45.854304    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"download-server\" with ErrImagePull: \"rpc error: code = Canceled desc = unable to pull image: copying system image from manifest list: copying config: context canceled\"" pod="openshift-console/downloads-6dd789c8b7-cchgk" podUID="e6a65e50-7411-4a9f-af62-92340babdbbd"
Nov 02 19:48:46.268973 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:46.268946    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerStarted","Data":"6828e754add268050b64ee9cdcd77601204bc082d95a6d7f23e6e6cf8dd98c31"}
Nov 02 19:48:46.269078 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:46.269037    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerStarted","Data":"fcc008987798c35e0f256995e0cca8f812f0cd3b338815e4b15fc95af93fb6dd"}
Nov 02 19:48:46.269078 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:46.269063    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerStarted","Data":"4b7df815ef75dda989ebf311d24ceda0e748939d88c2b72b79f19ceaf3f53fc5"}
Nov 02 19:48:46.350526 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:46.350494    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:47.274541 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:47.274478    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g" event={"ID":"92b5e340-4e43-45e1-85e3-a867fda1c9f6","Type":"ContainerStarted","Data":"9508f62822b058581272c796f65abfe5907276219b408b0cde18e5118a7f9d68"}
Nov 02 19:48:47.274919 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:47.274574    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g" event={"ID":"92b5e340-4e43-45e1-85e3-a867fda1c9f6","Type":"ContainerStarted","Data":"8039896451e5cf7123aca3646d69f1f1d97b7ef61055770fc274a47d86a3815b"}
Nov 02 19:48:47.274919 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:47.274606    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g" event={"ID":"92b5e340-4e43-45e1-85e3-a867fda1c9f6","Type":"ContainerStarted","Data":"ffdf74fa828dee8568e91556ee708fdb0cbf682e4bd24e9b881a38093f78626c"}
Nov 02 19:48:47.277561 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:47.277532    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerStarted","Data":"f88e028f941802ee6fff8ead15ede095a21b0a88bdcfa4369f9f2eb94ac05e5c"}
Nov 02 19:48:47.277663 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:47.277593    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerStarted","Data":"38a029031a33b1abd3e0d0bef38109f11ece6f2329da7604e3759c247917539a"}
Nov 02 19:48:47.496735 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:47.496666    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5dcb85b9b6-7xp5g" podStartSLOduration=2.771916074 podStartE2EDuration="5.496651927s" totalImagesPullingTime="2.724735853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:47.44549542 +0000 UTC m=+8.443184025" watchObservedRunningTime="2026-11-02 19:48:47.496651927 +0000 UTC m=+8.494340536"
Nov 02 19:48:48.282978 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:48.282944    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:48.284999 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:48.284975    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerStarted","Data":"72b4a966b54095fb3b2907c0d283d13774f6c9e572f24e7b033630bb61ae5ecc"}
Nov 02 19:48:48.479264 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:48.479170    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:48.494931 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:48.494884    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.5004849460000003 podStartE2EDuration="9.494870526s" totalImagesPullingTime="5.981270819s" totalInitContainerRuntime="13.114761ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:48.462062979 +0000 UTC m=+9.459751588" watchObservedRunningTime="2026-11-02 19:48:48.494870526 +0000 UTC m=+9.492559135"
Nov 02 19:48:48.659848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:48.659816    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:48.779566 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:48.779540    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbcmb"
Nov 02 19:48:48.832331 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:48.832286    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:49.006265 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:49.006235    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:49.010326 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:49.010306    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:49.010488 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:49.010477    5224 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 19:48:49.256656 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:49.256580    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4m6t"
Nov 02 19:48:49.380437 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:49.380383    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:49.964846 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:49.964805    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:49.969421 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:49.969396    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:50.295621 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:50.295594    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:48:52.337268 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:52.337238    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jwvp7"
Nov 02 19:48:53.667908 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:53.667880    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:53.883057 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:53.883017    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:54.202482 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:54.202430    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:54.390045 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:54.390011    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:56.475028 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.474935    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:48:56.547000 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.546969    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:48:56.578837 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.578814    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.598379 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.598354    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:48:56.648019 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.647983    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 19:48:56.648019 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.647984    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 19:48:56.648019 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.647984    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 19:48:56.648240 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.648048    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a6pl4uph7s6pt\""
Nov 02 19:48:56.648828 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.648811    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 19:48:56.648926 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.648835    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 19:48:56.648926 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.648858    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 19:48:56.648926 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.648835    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mvxpm\""
Nov 02 19:48:56.648926 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.648864    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 19:48:56.648926 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.648896    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 19:48:56.674594 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.674569    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm26r\" (UniqueName: \"kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-kube-api-access-xm26r\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.674702 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.674599    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.674702 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.674616    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.674702 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.674635    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.674828 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.674808    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.674863 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.674837    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.674896 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.674861    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.674927 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.674899    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.674959 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.674927    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.674959 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.674946    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.675118 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.674967    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.675118 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.674990    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.675118 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.675042    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.675118 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.675071    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.675295 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.675160    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.675295 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.675184    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.675295 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.675200    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.675295 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.675235    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.675429 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.675289    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.675429 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.675324    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.684100 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.684082    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 19:48:56.750692 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.750619    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 19:48:56.750692 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.750654    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 19:48:56.750692 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.750684    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 19:48:56.750966 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.750687    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 19:48:56.750966 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.750706    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 19:48:56.751698 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.751680    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 19:48:56.776286 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776256    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776401 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776299    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776401 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776325    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776401 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776354    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776401 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776387    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776412    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776443    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776475    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776506    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776600 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776570    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm26r\" (UniqueName: \"kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-kube-api-access-xm26r\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776862 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776600    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776862 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776656    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776862 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776691    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776862 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776736    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776862 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776759    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776862 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776814    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776862 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776837    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.776862 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776862    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.777364 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776887    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.777364 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.776922    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.777479 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.777425    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.777607 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.777486    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.777607 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.777593    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.779090 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.778339    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.779090 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.778423    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.779260 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.779238    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.779512 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.779493    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.780106 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.780084    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.781035 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.780631    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.781035 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.780658    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.781035 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.780990    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.781499 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.781353    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.781729 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.781704    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.781851 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.781748    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.781949 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.781927    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.782034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.782014    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.782495 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.782469    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.783015 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.782991    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.783611 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.783594    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.834949 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.834923    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm26r\" (UniqueName: \"kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-kube-api-access-xm26r\") pod \"prometheus-k8s-0\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:56.983429 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:56.983396    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:48:57.358623 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:48:57.358542    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:48:57.473338 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:48:57.473308    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c5786c_518d_46f7_98c7_1d33f1b0c5d2.slice/crio-0c46b4167cdc2f43c2b38900d940109ad3422093cb3b8752c3734ea0e388183b WatchSource:0}: Error finding container 0c46b4167cdc2f43c2b38900d940109ad3422093cb3b8752c3734ea0e388183b: Status 404 returned error can't find the container with id 0c46b4167cdc2f43c2b38900d940109ad3422093cb3b8752c3734ea0e388183b
Nov 02 19:48:57.476303 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:57.476277    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:48:58.321182 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:58.321136    5224 generic.go:356] "Generic (PLEG): container finished" podID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerID="54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038" exitCode=0
Nov 02 19:48:58.321354 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:58.321246    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerDied","Data":"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"}
Nov 02 19:48:58.321354 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:58.321350    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerStarted","Data":"0c46b4167cdc2f43c2b38900d940109ad3422093cb3b8752c3734ea0e388183b"}
Nov 02 19:48:58.544237 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:48:58.544199    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:00.488024 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:00.487984    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:02.344025 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:02.343967    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-cchgk" event={"ID":"e6a65e50-7411-4a9f-af62-92340babdbbd","Type":"ContainerStarted","Data":"7d134570d56da8ed45ff5cc916cbcc33be284872a937f954985e420399b8fc60"}
Nov 02 19:49:02.345046 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:02.345024    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:49:02.356198 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:02.356171    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-cchgk"
Nov 02 19:49:03.285322 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:03.285277    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:03.502332 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:03.502246    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:04.356038 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:04.356005    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerStarted","Data":"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"}
Nov 02 19:49:04.473095 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:04.472974    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:04.692697 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:04.692663    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:05.363948 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:05.363909    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerStarted","Data":"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"}
Nov 02 19:49:06.071484 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:06.071405    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:06.290555 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:06.290513    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:07.939602 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:07.939576    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-794cd4c8c5-td84z"]
Nov 02 19:49:08.383097 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:08.383058    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerStarted","Data":"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"}
Nov 02 19:49:08.383292 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:08.383142    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerStarted","Data":"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"}
Nov 02 19:49:08.383292 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:08.383182    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerStarted","Data":"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"}
Nov 02 19:49:08.383292 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:08.383220    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerStarted","Data":"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"}
Nov 02 19:49:08.609227 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:08.609167    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.5388704349999998 podStartE2EDuration="12.609151649s" totalImagesPullingTime="9.054456504s" totalInitContainerRuntime="15.82471ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:08.557795152 +0000 UTC m=+29.555483763" watchObservedRunningTime="2026-11-02 19:49:08.609151649 +0000 UTC m=+29.606840258"
Nov 02 19:49:09.394805 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:09.394745    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:11.983885 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:11.983842    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:49:13.909248 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:13.909217    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hxstf"
Nov 02 19:49:18.298403 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:18.298362    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:18.520432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:18.520398    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:27.356721 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:49:27.356688    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:49:32.961008 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:32.960962    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-794cd4c8c5-td84z" podUID="477caeb6-d7be-439d-a2e9-ce396eadb00b" containerName="console" containerID="cri-o://11d9f9816cd7150e36554e7fa82b104b975c95e11169181e469ee4d7af096dc5" gracePeriod=15
Nov 02 19:49:33.163214 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.163062    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:33.241876 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.241853    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:49:33.292209 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.292175    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-serving-cert\") pod \"477caeb6-d7be-439d-a2e9-ce396eadb00b\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") "
Nov 02 19:49:33.292397 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.292230    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-config\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-config\") pod \"477caeb6-d7be-439d-a2e9-ce396eadb00b\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") "
Nov 02 19:49:33.292397 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.292251    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477caeb6-d7be-439d-a2e9-ce396eadb00b-kube-api-access-fp4ls\" (UniqueName: \"kubernetes.io/projected/477caeb6-d7be-439d-a2e9-ce396eadb00b-kube-api-access-fp4ls\") pod \"477caeb6-d7be-439d-a2e9-ce396eadb00b\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") "
Nov 02 19:49:33.292397 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.292280    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-service-ca\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-service-ca\") pod \"477caeb6-d7be-439d-a2e9-ce396eadb00b\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") "
Nov 02 19:49:33.292397 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.292319    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477caeb6-d7be-439d-a2e9-ce396eadb00b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/477caeb6-d7be-439d-a2e9-ce396eadb00b-tmp\") pod \"477caeb6-d7be-439d-a2e9-ce396eadb00b\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") "
Nov 02 19:49:33.292397 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.292353    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-oauth-serving-cert\") pod \"477caeb6-d7be-439d-a2e9-ce396eadb00b\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") "
Nov 02 19:49:33.292397 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.292397    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-oauth-config\") pod \"477caeb6-d7be-439d-a2e9-ce396eadb00b\" (UID: \"477caeb6-d7be-439d-a2e9-ce396eadb00b\") "
Nov 02 19:49:33.292690 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.292644    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477caeb6-d7be-439d-a2e9-ce396eadb00b-tmp" pod "477caeb6-d7be-439d-a2e9-ce396eadb00b" (UID: "477caeb6-d7be-439d-a2e9-ce396eadb00b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:49:33.292804 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.292748    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-service-ca" pod "477caeb6-d7be-439d-a2e9-ce396eadb00b" (UID: "477caeb6-d7be-439d-a2e9-ce396eadb00b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:49:33.292804 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.292759    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-oauth-serving-cert" pod "477caeb6-d7be-439d-a2e9-ce396eadb00b" (UID: "477caeb6-d7be-439d-a2e9-ce396eadb00b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:49:33.292987 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.292878    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-config" pod "477caeb6-d7be-439d-a2e9-ce396eadb00b" (UID: "477caeb6-d7be-439d-a2e9-ce396eadb00b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:49:33.294699 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.294669    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-oauth-config" pod "477caeb6-d7be-439d-a2e9-ce396eadb00b" (UID: "477caeb6-d7be-439d-a2e9-ce396eadb00b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:49:33.294793 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.294707    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477caeb6-d7be-439d-a2e9-ce396eadb00b-kube-api-access-fp4ls" pod "477caeb6-d7be-439d-a2e9-ce396eadb00b" (UID: "477caeb6-d7be-439d-a2e9-ce396eadb00b"). InnerVolumeSpecName "kube-api-access-fp4ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:49:33.294793 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.294736    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-serving-cert" pod "477caeb6-d7be-439d-a2e9-ce396eadb00b" (UID: "477caeb6-d7be-439d-a2e9-ce396eadb00b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:49:33.393073 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.393040    5224 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-serving-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:49:33.393073 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.393067    5224 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-config\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:49:33.393073 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.393077    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp4ls\" (UniqueName: \"kubernetes.io/projected/477caeb6-d7be-439d-a2e9-ce396eadb00b-kube-api-access-fp4ls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:49:33.393290 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.393087    5224 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:49:33.393290 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.393097    5224 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/477caeb6-d7be-439d-a2e9-ce396eadb00b-tmp\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:49:33.393290 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.393107    5224 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/477caeb6-d7be-439d-a2e9-ce396eadb00b-oauth-serving-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:49:33.393290 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.393115    5224 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/477caeb6-d7be-439d-a2e9-ce396eadb00b-console-oauth-config\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:49:33.477137 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.477056    5224 generic.go:356] "Generic (PLEG): container finished" podID="477caeb6-d7be-439d-a2e9-ce396eadb00b" containerID="11d9f9816cd7150e36554e7fa82b104b975c95e11169181e469ee4d7af096dc5" exitCode=0
Nov 02 19:49:33.477137 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.477101    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794cd4c8c5-td84z"
Nov 02 19:49:33.477321 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.477143    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794cd4c8c5-td84z" event={"ID":"477caeb6-d7be-439d-a2e9-ce396eadb00b","Type":"ContainerDied","Data":"11d9f9816cd7150e36554e7fa82b104b975c95e11169181e469ee4d7af096dc5"}
Nov 02 19:49:33.477321 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.477202    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794cd4c8c5-td84z" event={"ID":"477caeb6-d7be-439d-a2e9-ce396eadb00b","Type":"ContainerDied","Data":"6c0ad274a9c56c5e2e0c307ed7fbb52181e229fd94bd975fbc869cf14894cc4b"}
Nov 02 19:49:33.477321 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.477226    5224 scope.go:122] "RemoveContainer" containerID="11d9f9816cd7150e36554e7fa82b104b975c95e11169181e469ee4d7af096dc5"
Nov 02 19:49:33.485566 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.485549    5224 scope.go:122] "RemoveContainer" containerID="11d9f9816cd7150e36554e7fa82b104b975c95e11169181e469ee4d7af096dc5"
Nov 02 19:49:33.485854 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:49:33.485834    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d9f9816cd7150e36554e7fa82b104b975c95e11169181e469ee4d7af096dc5\": container with ID starting with 11d9f9816cd7150e36554e7fa82b104b975c95e11169181e469ee4d7af096dc5 not found: ID does not exist" containerID="11d9f9816cd7150e36554e7fa82b104b975c95e11169181e469ee4d7af096dc5"
Nov 02 19:49:33.485951 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.485864    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d9f9816cd7150e36554e7fa82b104b975c95e11169181e469ee4d7af096dc5"} err="failed to get container status \"11d9f9816cd7150e36554e7fa82b104b975c95e11169181e469ee4d7af096dc5\": rpc error: code = NotFound desc = could not find container \"11d9f9816cd7150e36554e7fa82b104b975c95e11169181e469ee4d7af096dc5\": container with ID starting with 11d9f9816cd7150e36554e7fa82b104b975c95e11169181e469ee4d7af096dc5 not found: ID does not exist"
Nov 02 19:49:33.643861 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.643831    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-794cd4c8c5-td84z"]
Nov 02 19:49:33.647850 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.647831    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-794cd4c8c5-td84z"]
Nov 02 19:49:33.823818 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:33.823758    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:35.165041 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:35.165002    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="477caeb6-d7be-439d-a2e9-ce396eadb00b" path="/var/lib/kubelet/pods/477caeb6-d7be-439d-a2e9-ce396eadb00b/volumes"
Nov 02 19:49:44.991121 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:44.991086    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:46.104781 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:46.104722    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:49:57.356936 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:49:57.356892    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:49:57.368641 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:57.368626    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:49:57.386910 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:57.386882    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:49:57.574973 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:49:57.574944    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:06.338212 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:06.338171    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:50:14.720407 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:14.720366    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:50:14.946867 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:14.946834    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:50:15.116840 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:15.116796    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:50:18.979724 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:18.979620    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:50:19.200897 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:19.200853    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:50:24.570367 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:24.570334    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 19:50:24.570816 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:24.570620    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="alertmanager" containerID="cri-o://4b7df815ef75dda989ebf311d24ceda0e748939d88c2b72b79f19ceaf3f53fc5" gracePeriod=120
Nov 02 19:50:24.570816 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:24.570690    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="kube-rbac-proxy-metric" containerID="cri-o://f88e028f941802ee6fff8ead15ede095a21b0a88bdcfa4369f9f2eb94ac05e5c" gracePeriod=120
Nov 02 19:50:24.570816 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:24.570742    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="prom-label-proxy" containerID="cri-o://72b4a966b54095fb3b2907c0d283d13774f6c9e572f24e7b033630bb61ae5ecc" gracePeriod=120
Nov 02 19:50:24.570816 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:24.570696    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="kube-rbac-proxy-web" containerID="cri-o://6828e754add268050b64ee9cdcd77601204bc082d95a6d7f23e6e6cf8dd98c31" gracePeriod=120
Nov 02 19:50:24.571030 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:24.570789    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="config-reloader" containerID="cri-o://fcc008987798c35e0f256995e0cca8f812f0cd3b338815e4b15fc95af93fb6dd" gracePeriod=120
Nov 02 19:50:24.571030 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:24.570725    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="kube-rbac-proxy" containerID="cri-o://38a029031a33b1abd3e0d0bef38109f11ece6f2329da7604e3759c247917539a" gracePeriod=120
Nov 02 19:50:25.662637 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.662602    5224 generic.go:356] "Generic (PLEG): container finished" podID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerID="72b4a966b54095fb3b2907c0d283d13774f6c9e572f24e7b033630bb61ae5ecc" exitCode=0
Nov 02 19:50:25.662637 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.662628    5224 generic.go:356] "Generic (PLEG): container finished" podID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerID="38a029031a33b1abd3e0d0bef38109f11ece6f2329da7604e3759c247917539a" exitCode=0
Nov 02 19:50:25.662637 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.662634    5224 generic.go:356] "Generic (PLEG): container finished" podID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerID="6828e754add268050b64ee9cdcd77601204bc082d95a6d7f23e6e6cf8dd98c31" exitCode=0
Nov 02 19:50:25.662637 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.662642    5224 generic.go:356] "Generic (PLEG): container finished" podID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerID="fcc008987798c35e0f256995e0cca8f812f0cd3b338815e4b15fc95af93fb6dd" exitCode=0
Nov 02 19:50:25.663171 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.662651    5224 generic.go:356] "Generic (PLEG): container finished" podID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerID="4b7df815ef75dda989ebf311d24ceda0e748939d88c2b72b79f19ceaf3f53fc5" exitCode=0
Nov 02 19:50:25.663171 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.662672    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerDied","Data":"72b4a966b54095fb3b2907c0d283d13774f6c9e572f24e7b033630bb61ae5ecc"}
Nov 02 19:50:25.663171 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.662741    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerDied","Data":"38a029031a33b1abd3e0d0bef38109f11ece6f2329da7604e3759c247917539a"}
Nov 02 19:50:25.663171 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.662780    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerDied","Data":"6828e754add268050b64ee9cdcd77601204bc082d95a6d7f23e6e6cf8dd98c31"}
Nov 02 19:50:25.663171 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.662808    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerDied","Data":"fcc008987798c35e0f256995e0cca8f812f0cd3b338815e4b15fc95af93fb6dd"}
Nov 02 19:50:25.663171 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.662831    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerDied","Data":"4b7df815ef75dda989ebf311d24ceda0e748939d88c2b72b79f19ceaf3f53fc5"}
Nov 02 19:50:25.819636 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.819608    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:25.921762 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.921664    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") "
Nov 02 19:50:25.921762 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.921715    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-main-db\") pod \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") "
Nov 02 19:50:25.921762 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.921746    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-cluster-tls-config\") pod \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") "
Nov 02 19:50:25.922087 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.921800    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-web\") pod \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") "
Nov 02 19:50:25.922087 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.921845    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy\") pod \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") "
Nov 02 19:50:25.922087 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.921878    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-main-tls\") pod \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") "
Nov 02 19:50:25.922087 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.921945    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-web-config\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-web-config\") pod \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") "
Nov 02 19:50:25.922087 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.921976    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-metrics-client-ca\") pod \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") "
Nov 02 19:50:25.922349 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.922192    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-tls-assets\" (UniqueName: \"kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-tls-assets\") pod \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") "
Nov 02 19:50:25.922349 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.922264    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-volume\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-volume\") pod \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") "
Nov 02 19:50:25.922349 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.922298    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-kube-api-access-989mm\" (UniqueName: \"kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-kube-api-access-989mm\") pod \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") "
Nov 02 19:50:25.922349 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.922347    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-out\") pod \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") "
Nov 02 19:50:25.922553 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.922378    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-trusted-ca-bundle\") pod \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\" (UID: \"a7d78e17-113c-4787-8a1d-5ead8f950bfc\") "
Nov 02 19:50:25.922630 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.922598    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-metrics-client-ca" pod "a7d78e17-113c-4787-8a1d-5ead8f950bfc" (UID: "a7d78e17-113c-4787-8a1d-5ead8f950bfc"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:50:25.922982 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.922718    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-main-db" pod "a7d78e17-113c-4787-8a1d-5ead8f950bfc" (UID: "a7d78e17-113c-4787-8a1d-5ead8f950bfc"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:50:25.923126 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.922986    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-trusted-ca-bundle" pod "a7d78e17-113c-4787-8a1d-5ead8f950bfc" (UID: "a7d78e17-113c-4787-8a1d-5ead8f950bfc"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:50:25.924964 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.924927    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy" pod "a7d78e17-113c-4787-8a1d-5ead8f950bfc" (UID: "a7d78e17-113c-4787-8a1d-5ead8f950bfc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:25.925224 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.925181    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-web" pod "a7d78e17-113c-4787-8a1d-5ead8f950bfc" (UID: "a7d78e17-113c-4787-8a1d-5ead8f950bfc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:25.925912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.925875    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-kube-api-access-989mm" pod "a7d78e17-113c-4787-8a1d-5ead8f950bfc" (UID: "a7d78e17-113c-4787-8a1d-5ead8f950bfc"). InnerVolumeSpecName "kube-api-access-989mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:50:25.925912 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.925896    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-metric" pod "a7d78e17-113c-4787-8a1d-5ead8f950bfc" (UID: "a7d78e17-113c-4787-8a1d-5ead8f950bfc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:25.926117 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.926022    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-main-tls" pod "a7d78e17-113c-4787-8a1d-5ead8f950bfc" (UID: "a7d78e17-113c-4787-8a1d-5ead8f950bfc"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:25.926190 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.926170    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-volume" pod "a7d78e17-113c-4787-8a1d-5ead8f950bfc" (UID: "a7d78e17-113c-4787-8a1d-5ead8f950bfc"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:25.926334 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.926305    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-out" pod "a7d78e17-113c-4787-8a1d-5ead8f950bfc" (UID: "a7d78e17-113c-4787-8a1d-5ead8f950bfc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:50:25.926977 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.926956    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-tls-assets" pod "a7d78e17-113c-4787-8a1d-5ead8f950bfc" (UID: "a7d78e17-113c-4787-8a1d-5ead8f950bfc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:50:25.929969 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.929790    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-cluster-tls-config" pod "a7d78e17-113c-4787-8a1d-5ead8f950bfc" (UID: "a7d78e17-113c-4787-8a1d-5ead8f950bfc"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:25.936199 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:25.936169    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-web-config" pod "a7d78e17-113c-4787-8a1d-5ead8f950bfc" (UID: "a7d78e17-113c-4787-8a1d-5ead8f950bfc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:26.023930 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.023884    5224 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:26.023930 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.023923    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-989mm\" (UniqueName: \"kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-kube-api-access-989mm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:26.023930 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.023941    5224 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-config-out\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:26.024200 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.023954    5224 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:26.024200 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.023968    5224 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:26.024200 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.023981    5224 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a7d78e17-113c-4787-8a1d-5ead8f950bfc-alertmanager-main-db\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:26.024200 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.023993    5224 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-cluster-tls-config\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:26.024200 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.024005    5224 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:26.024200 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.024017    5224 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:26.024200 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.024030    5224 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-secret-alertmanager-main-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:26.024200 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.024042    5224 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7d78e17-113c-4787-8a1d-5ead8f950bfc-web-config\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:26.024200 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.024052    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7d78e17-113c-4787-8a1d-5ead8f950bfc-metrics-client-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:26.024200 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.024067    5224 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7d78e17-113c-4787-8a1d-5ead8f950bfc-tls-assets\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:26.669944 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.669914    5224 generic.go:356] "Generic (PLEG): container finished" podID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerID="f88e028f941802ee6fff8ead15ede095a21b0a88bdcfa4369f9f2eb94ac05e5c" exitCode=0
Nov 02 19:50:26.670331 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.669982    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerDied","Data":"f88e028f941802ee6fff8ead15ede095a21b0a88bdcfa4369f9f2eb94ac05e5c"}
Nov 02 19:50:26.670331 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.670012    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:26.670331 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.670039    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7d78e17-113c-4787-8a1d-5ead8f950bfc","Type":"ContainerDied","Data":"36dd88251280c84e26c5153af2cd5790542a9afd8b2a17fedddb4ee56ec2ab59"}
Nov 02 19:50:26.670331 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.670073    5224 scope.go:122] "RemoveContainer" containerID="72b4a966b54095fb3b2907c0d283d13774f6c9e572f24e7b033630bb61ae5ecc"
Nov 02 19:50:26.679543 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.679523    5224 scope.go:122] "RemoveContainer" containerID="f88e028f941802ee6fff8ead15ede095a21b0a88bdcfa4369f9f2eb94ac05e5c"
Nov 02 19:50:26.686883 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.686700    5224 scope.go:122] "RemoveContainer" containerID="38a029031a33b1abd3e0d0bef38109f11ece6f2329da7604e3759c247917539a"
Nov 02 19:50:26.693628 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.693611    5224 scope.go:122] "RemoveContainer" containerID="6828e754add268050b64ee9cdcd77601204bc082d95a6d7f23e6e6cf8dd98c31"
Nov 02 19:50:26.700634 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.700613    5224 scope.go:122] "RemoveContainer" containerID="fcc008987798c35e0f256995e0cca8f812f0cd3b338815e4b15fc95af93fb6dd"
Nov 02 19:50:26.708051 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.708029    5224 scope.go:122] "RemoveContainer" containerID="4b7df815ef75dda989ebf311d24ceda0e748939d88c2b72b79f19ceaf3f53fc5"
Nov 02 19:50:26.715339 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.715318    5224 scope.go:122] "RemoveContainer" containerID="e82bbc55b9369caa40791b5668890d12e14fae0651c19ef70ec21471a2c548e5"
Nov 02 19:50:26.722540 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.722522    5224 scope.go:122] "RemoveContainer" containerID="72b4a966b54095fb3b2907c0d283d13774f6c9e572f24e7b033630bb61ae5ecc"
Nov 02 19:50:26.722865 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:26.722843    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b4a966b54095fb3b2907c0d283d13774f6c9e572f24e7b033630bb61ae5ecc\": container with ID starting with 72b4a966b54095fb3b2907c0d283d13774f6c9e572f24e7b033630bb61ae5ecc not found: ID does not exist" containerID="72b4a966b54095fb3b2907c0d283d13774f6c9e572f24e7b033630bb61ae5ecc"
Nov 02 19:50:26.722924 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.722877    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b4a966b54095fb3b2907c0d283d13774f6c9e572f24e7b033630bb61ae5ecc"} err="failed to get container status \"72b4a966b54095fb3b2907c0d283d13774f6c9e572f24e7b033630bb61ae5ecc\": rpc error: code = NotFound desc = could not find container \"72b4a966b54095fb3b2907c0d283d13774f6c9e572f24e7b033630bb61ae5ecc\": container with ID starting with 72b4a966b54095fb3b2907c0d283d13774f6c9e572f24e7b033630bb61ae5ecc not found: ID does not exist"
Nov 02 19:50:26.722924 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.722896    5224 scope.go:122] "RemoveContainer" containerID="f88e028f941802ee6fff8ead15ede095a21b0a88bdcfa4369f9f2eb94ac05e5c"
Nov 02 19:50:26.723190 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:26.723171    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88e028f941802ee6fff8ead15ede095a21b0a88bdcfa4369f9f2eb94ac05e5c\": container with ID starting with f88e028f941802ee6fff8ead15ede095a21b0a88bdcfa4369f9f2eb94ac05e5c not found: ID does not exist" containerID="f88e028f941802ee6fff8ead15ede095a21b0a88bdcfa4369f9f2eb94ac05e5c"
Nov 02 19:50:26.723266 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.723197    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88e028f941802ee6fff8ead15ede095a21b0a88bdcfa4369f9f2eb94ac05e5c"} err="failed to get container status \"f88e028f941802ee6fff8ead15ede095a21b0a88bdcfa4369f9f2eb94ac05e5c\": rpc error: code = NotFound desc = could not find container \"f88e028f941802ee6fff8ead15ede095a21b0a88bdcfa4369f9f2eb94ac05e5c\": container with ID starting with f88e028f941802ee6fff8ead15ede095a21b0a88bdcfa4369f9f2eb94ac05e5c not found: ID does not exist"
Nov 02 19:50:26.723266 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.723215    5224 scope.go:122] "RemoveContainer" containerID="38a029031a33b1abd3e0d0bef38109f11ece6f2329da7604e3759c247917539a"
Nov 02 19:50:26.723467 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:26.723449    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a029031a33b1abd3e0d0bef38109f11ece6f2329da7604e3759c247917539a\": container with ID starting with 38a029031a33b1abd3e0d0bef38109f11ece6f2329da7604e3759c247917539a not found: ID does not exist" containerID="38a029031a33b1abd3e0d0bef38109f11ece6f2329da7604e3759c247917539a"
Nov 02 19:50:26.723503 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.723471    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a029031a33b1abd3e0d0bef38109f11ece6f2329da7604e3759c247917539a"} err="failed to get container status \"38a029031a33b1abd3e0d0bef38109f11ece6f2329da7604e3759c247917539a\": rpc error: code = NotFound desc = could not find container \"38a029031a33b1abd3e0d0bef38109f11ece6f2329da7604e3759c247917539a\": container with ID starting with 38a029031a33b1abd3e0d0bef38109f11ece6f2329da7604e3759c247917539a not found: ID does not exist"
Nov 02 19:50:26.723503 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.723485    5224 scope.go:122] "RemoveContainer" containerID="6828e754add268050b64ee9cdcd77601204bc082d95a6d7f23e6e6cf8dd98c31"
Nov 02 19:50:26.723711 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:26.723696    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6828e754add268050b64ee9cdcd77601204bc082d95a6d7f23e6e6cf8dd98c31\": container with ID starting with 6828e754add268050b64ee9cdcd77601204bc082d95a6d7f23e6e6cf8dd98c31 not found: ID does not exist" containerID="6828e754add268050b64ee9cdcd77601204bc082d95a6d7f23e6e6cf8dd98c31"
Nov 02 19:50:26.723749 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.723715    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6828e754add268050b64ee9cdcd77601204bc082d95a6d7f23e6e6cf8dd98c31"} err="failed to get container status \"6828e754add268050b64ee9cdcd77601204bc082d95a6d7f23e6e6cf8dd98c31\": rpc error: code = NotFound desc = could not find container \"6828e754add268050b64ee9cdcd77601204bc082d95a6d7f23e6e6cf8dd98c31\": container with ID starting with 6828e754add268050b64ee9cdcd77601204bc082d95a6d7f23e6e6cf8dd98c31 not found: ID does not exist"
Nov 02 19:50:26.723749 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.723727    5224 scope.go:122] "RemoveContainer" containerID="fcc008987798c35e0f256995e0cca8f812f0cd3b338815e4b15fc95af93fb6dd"
Nov 02 19:50:26.723977 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:26.723958    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc008987798c35e0f256995e0cca8f812f0cd3b338815e4b15fc95af93fb6dd\": container with ID starting with fcc008987798c35e0f256995e0cca8f812f0cd3b338815e4b15fc95af93fb6dd not found: ID does not exist" containerID="fcc008987798c35e0f256995e0cca8f812f0cd3b338815e4b15fc95af93fb6dd"
Nov 02 19:50:26.724012 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.723980    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc008987798c35e0f256995e0cca8f812f0cd3b338815e4b15fc95af93fb6dd"} err="failed to get container status \"fcc008987798c35e0f256995e0cca8f812f0cd3b338815e4b15fc95af93fb6dd\": rpc error: code = NotFound desc = could not find container \"fcc008987798c35e0f256995e0cca8f812f0cd3b338815e4b15fc95af93fb6dd\": container with ID starting with fcc008987798c35e0f256995e0cca8f812f0cd3b338815e4b15fc95af93fb6dd not found: ID does not exist"
Nov 02 19:50:26.724012 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.723995    5224 scope.go:122] "RemoveContainer" containerID="4b7df815ef75dda989ebf311d24ceda0e748939d88c2b72b79f19ceaf3f53fc5"
Nov 02 19:50:26.724207 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:26.724192    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7df815ef75dda989ebf311d24ceda0e748939d88c2b72b79f19ceaf3f53fc5\": container with ID starting with 4b7df815ef75dda989ebf311d24ceda0e748939d88c2b72b79f19ceaf3f53fc5 not found: ID does not exist" containerID="4b7df815ef75dda989ebf311d24ceda0e748939d88c2b72b79f19ceaf3f53fc5"
Nov 02 19:50:26.724242 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.724215    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7df815ef75dda989ebf311d24ceda0e748939d88c2b72b79f19ceaf3f53fc5"} err="failed to get container status \"4b7df815ef75dda989ebf311d24ceda0e748939d88c2b72b79f19ceaf3f53fc5\": rpc error: code = NotFound desc = could not find container \"4b7df815ef75dda989ebf311d24ceda0e748939d88c2b72b79f19ceaf3f53fc5\": container with ID starting with 4b7df815ef75dda989ebf311d24ceda0e748939d88c2b72b79f19ceaf3f53fc5 not found: ID does not exist"
Nov 02 19:50:26.724242 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.724229    5224 scope.go:122] "RemoveContainer" containerID="e82bbc55b9369caa40791b5668890d12e14fae0651c19ef70ec21471a2c548e5"
Nov 02 19:50:26.724448 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:26.724433    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e82bbc55b9369caa40791b5668890d12e14fae0651c19ef70ec21471a2c548e5\": container with ID starting with e82bbc55b9369caa40791b5668890d12e14fae0651c19ef70ec21471a2c548e5 not found: ID does not exist" containerID="e82bbc55b9369caa40791b5668890d12e14fae0651c19ef70ec21471a2c548e5"
Nov 02 19:50:26.724514 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.724452    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e82bbc55b9369caa40791b5668890d12e14fae0651c19ef70ec21471a2c548e5"} err="failed to get container status \"e82bbc55b9369caa40791b5668890d12e14fae0651c19ef70ec21471a2c548e5\": rpc error: code = NotFound desc = could not find container \"e82bbc55b9369caa40791b5668890d12e14fae0651c19ef70ec21471a2c548e5\": container with ID starting with e82bbc55b9369caa40791b5668890d12e14fae0651c19ef70ec21471a2c548e5 not found: ID does not exist"
Nov 02 19:50:26.891147 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.891108    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 19:50:26.942342 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.942253    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 19:50:26.942732 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.942714    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 19:50:26.943000 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.942986    5224 cpu_manager.go:414] "containerMap: removing container" podUID="477caeb6-d7be-439d-a2e9-ce396eadb00b" containerName="console"
Nov 02 19:50:26.943062 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943001    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="477caeb6-d7be-439d-a2e9-ce396eadb00b" containerName="console"
Nov 02 19:50:26.943062 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943019    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="config-reloader"
Nov 02 19:50:26.943062 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943025    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="config-reloader"
Nov 02 19:50:26.943062 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943035    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="kube-rbac-proxy"
Nov 02 19:50:26.943062 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943040    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="kube-rbac-proxy"
Nov 02 19:50:26.943062 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943051    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="kube-rbac-proxy-web"
Nov 02 19:50:26.943062 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943056    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="kube-rbac-proxy-web"
Nov 02 19:50:26.943062 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943066    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="kube-rbac-proxy-metric"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943070    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="kube-rbac-proxy-metric"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943081    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="init-config-reloader"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943086    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="init-config-reloader"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943094    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="alertmanager"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943098    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="alertmanager"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943109    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="prom-label-proxy"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943113    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="prom-label-proxy"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943154    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="prom-label-proxy"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943160    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="477caeb6-d7be-439d-a2e9-ce396eadb00b" containerName="console"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943166    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="config-reloader"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943171    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="kube-rbac-proxy"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943178    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="kube-rbac-proxy-web"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943183    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="kube-rbac-proxy-metric"
Nov 02 19:50:26.943325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.943189    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" containerName="alertmanager"
Nov 02 19:50:26.948854 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:26.948822    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.003384 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.003347    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 19:50:27.003384 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.003370    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 19:50:27.003384 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.003350    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 19:50:27.003384 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.003361    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 19:50:27.003723 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.003422    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 19:50:27.003723 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.003478    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 19:50:27.003723 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.003574    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 19:50:27.003723 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.003579    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 19:50:27.003723 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.003599    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ww82t\""
Nov 02 19:50:27.031693 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.031651    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.031901 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.031724    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.031901 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.031800    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5fqg\" (UniqueName: \"kubernetes.io/projected/18474534-444d-45ea-8a53-20cace7aa3e1-kube-api-access-h5fqg\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.031901 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.031890    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-web-config\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.032085 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.031950    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.032085 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.031993    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18474534-444d-45ea-8a53-20cace7aa3e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.032085 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.032047    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18474534-444d-45ea-8a53-20cace7aa3e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.032206 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.032084    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.032206 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.032117    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18474534-444d-45ea-8a53-20cace7aa3e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.032206 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.032146    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.032206 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.032195    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.032321 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.032219    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18474534-444d-45ea-8a53-20cace7aa3e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.032321 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.032246    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18474534-444d-45ea-8a53-20cace7aa3e1-config-out\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.054915 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.054875    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 19:50:27.133103 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133059    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18474534-444d-45ea-8a53-20cace7aa3e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.133103 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133105    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18474534-444d-45ea-8a53-20cace7aa3e1-config-out\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.133355 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133135    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.133355 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133169    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.133355 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133186    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5fqg\" (UniqueName: \"kubernetes.io/projected/18474534-444d-45ea-8a53-20cace7aa3e1-kube-api-access-h5fqg\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.133355 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133205    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-web-config\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.133355 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133225    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.133355 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133256    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18474534-444d-45ea-8a53-20cace7aa3e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.133355 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133283    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18474534-444d-45ea-8a53-20cace7aa3e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.133355 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133314    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.133355 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133342    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18474534-444d-45ea-8a53-20cace7aa3e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.133817 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133368    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.133817 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133395    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.133996 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.133969    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18474534-444d-45ea-8a53-20cace7aa3e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.134282 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.134239    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18474534-444d-45ea-8a53-20cace7aa3e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.136458 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.136428    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-web-config\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.136569 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.136431    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.136990 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.136715    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18474534-444d-45ea-8a53-20cace7aa3e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.136990 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.136791    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.136990 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.136806    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.136990 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.136857    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.136990 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.136926    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18474534-444d-45ea-8a53-20cace7aa3e1-config-out\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.137264 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.137080    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18474534-444d-45ea-8a53-20cace7aa3e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.137549 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.137528    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.138035 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.138016    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18474534-444d-45ea-8a53-20cace7aa3e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.153666 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.153636    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 19:50:27.165461 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.165431    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d78e17-113c-4787-8a1d-5ead8f950bfc" path="/var/lib/kubelet/pods/a7d78e17-113c-4787-8a1d-5ead8f950bfc/volumes"
Nov 02 19:50:27.191469 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.191435    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5fqg\" (UniqueName: \"kubernetes.io/projected/18474534-444d-45ea-8a53-20cace7aa3e1-kube-api-access-h5fqg\") pod \"alertmanager-main-0\" (UID: \"18474534-444d-45ea-8a53-20cace7aa3e1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.265357 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.265322    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 19:50:27.357914 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:27.357880    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:50:27.679539 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:50:27.679510    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18474534_444d_45ea_8a53_20cace7aa3e1.slice/crio-1d8271cb15f9458cc5cf1234d803291ec3be85ed96039d900b2c3e03e9e5ef95 WatchSource:0}: Error finding container 1d8271cb15f9458cc5cf1234d803291ec3be85ed96039d900b2c3e03e9e5ef95: Status 404 returned error can't find the container with id 1d8271cb15f9458cc5cf1234d803291ec3be85ed96039d900b2c3e03e9e5ef95
Nov 02 19:50:27.681257 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:27.681236    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 19:50:28.680898 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:28.680859    5224 generic.go:356] "Generic (PLEG): container finished" podID="18474534-444d-45ea-8a53-20cace7aa3e1" containerID="be61dc9c233b393599923bdd30e54ac1bbe9840ade72f9efcbffd10898e5bc27" exitCode=0
Nov 02 19:50:28.681274 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:28.680907    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18474534-444d-45ea-8a53-20cace7aa3e1","Type":"ContainerDied","Data":"be61dc9c233b393599923bdd30e54ac1bbe9840ade72f9efcbffd10898e5bc27"}
Nov 02 19:50:28.681274 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:28.680958    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18474534-444d-45ea-8a53-20cace7aa3e1","Type":"ContainerStarted","Data":"1d8271cb15f9458cc5cf1234d803291ec3be85ed96039d900b2c3e03e9e5ef95"}
Nov 02 19:50:29.323617 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:29.323575    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:50:29.494196 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:29.494157    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:50:29.687886 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:29.687792    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18474534-444d-45ea-8a53-20cace7aa3e1","Type":"ContainerStarted","Data":"4a0f38923da9c4158dbb31431e2e7f6310c857583da08c5399d800506e7c02af"}
Nov 02 19:50:29.687886 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:29.687870    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18474534-444d-45ea-8a53-20cace7aa3e1","Type":"ContainerStarted","Data":"3732aee928f0f74cc10844ab97bf74b061f484d02b1378b2ad1e76c4428abc8f"}
Nov 02 19:50:29.688422 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:29.687912    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18474534-444d-45ea-8a53-20cace7aa3e1","Type":"ContainerStarted","Data":"83eed4cdaf5b41c6d8be3f1ce34637a7ab69fe0dbe68299bb16f7b73408826e2"}
Nov 02 19:50:29.688422 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:29.687951    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18474534-444d-45ea-8a53-20cace7aa3e1","Type":"ContainerStarted","Data":"35a0d405b5dda44652cdeee1506586469999b94120b3ae52df2b58f132eab27a"}
Nov 02 19:50:29.688422 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:29.687995    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18474534-444d-45ea-8a53-20cace7aa3e1","Type":"ContainerStarted","Data":"c83b8e146438e855d35b2b4f396808344757549d86ba145830424c41b730fd77"}
Nov 02 19:50:29.688422 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:29.688036    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18474534-444d-45ea-8a53-20cace7aa3e1","Type":"ContainerStarted","Data":"7ed1b7a2b2c33234beeae621c38ea3ff9a8f89559fa8ab86899ac259306efad0"}
Nov 02 19:50:29.814199 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:29.814138    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.801248751 podStartE2EDuration="3.814122605s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.873854ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:50:29.812897613 +0000 UTC m=+110.810586224" watchObservedRunningTime="2026-11-02 19:50:29.814122605 +0000 UTC m=+110.811811213"
Nov 02 19:50:32.106688 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.106649    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:50:32.107224 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.106928    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="prometheus" containerID="cri-o://19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a" gracePeriod=600
Nov 02 19:50:32.107224 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.106950    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="kube-rbac-proxy" containerID="cri-o://babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4" gracePeriod=600
Nov 02 19:50:32.107224 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.106956    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="config-reloader" containerID="cri-o://5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156" gracePeriod=600
Nov 02 19:50:32.107224 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.106981    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7" gracePeriod=600
Nov 02 19:50:32.107224 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.107004    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="kube-rbac-proxy-web" containerID="cri-o://298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d" gracePeriod=600
Nov 02 19:50:32.107224 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.106956    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="thanos-sidecar" containerID="cri-o://20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1" gracePeriod=600
Nov 02 19:50:32.354521 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.354492    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.375548 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.375471    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.375548 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.375517    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-tls-assets\" (UniqueName: \"kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-tls-assets\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.375548 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.375537    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-web-config\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-web-config\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.375749 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.375572    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-grpc-tls\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.375749 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.375599    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-serving-certs-ca-bundle\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.375749 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.375634    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-2\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.375749 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.375668    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config-out\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376013    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-trusted-ca-bundle\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376049    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-2" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376082    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-tls\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376100    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-serving-certs-ca-bundle" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376119    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376154    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-0\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376193    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-kube-rbac-proxy\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376220    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-db\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376249    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-metrics-client-certs\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376284    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-kube-api-access-xm26r\" (UniqueName: \"kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-kube-api-access-xm26r\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376321    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-thanos-prometheus-http-client-file\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376346    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-1\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376381    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376414    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-trusted-ca-bundle" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376427    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-metrics-client-ca\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376459    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-kubelet-serving-ca-bundle\") pod \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\" (UID: \"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2\") "
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376696    5224 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376716    5224 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.376965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.376730    5224 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.378673 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.377966    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-kubelet-serving-ca-bundle" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:50:32.378673 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.378256    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-1" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:50:32.378673 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.378564    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-thanos-sidecar-tls" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:32.380057 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.379106    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config-out" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:50:32.380057 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.379384    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-tls" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:32.380057 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.379501    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-0" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:50:32.380057 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.379542    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-grpc-tls" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:32.380057 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.379796    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-db" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:50:32.380057 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.379818    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-metrics-client-ca" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:50:32.380057 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.379957    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-tls-assets" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:50:32.380496 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.380207    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-metrics-client-certs" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:32.381195 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.381173    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-thanos-prometheus-http-client-file" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:32.381666 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.381639    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-kube-rbac-proxy-web" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:32.381818 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.381796    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-kube-rbac-proxy" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:32.382570 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.382547    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:32.382671 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.382548    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-kube-api-access-xm26r" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "kube-api-access-xm26r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:50:32.389241 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.389218    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-web-config" pod "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" (UID: "f2c5786c-518d-46f7-98c7-1d33f1b0c5d2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:50:32.477813 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477756    5224 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-kube-rbac-proxy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477840    5224 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-db\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477853    5224 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-metrics-client-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477862    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm26r\" (UniqueName: \"kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-kube-api-access-xm26r\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477872    5224 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477883    5224 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477894    5224 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477902    5224 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-metrics-client-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477912    5224 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477922    5224 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477930    5224 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-tls-assets\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477940    5224 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-web-config\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477953    5224 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-grpc-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477962    5224 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-config-out\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477970    5224 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477979    5224 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.477993 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.477988    5224 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:50:32.700888 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.700795    5224 generic.go:356] "Generic (PLEG): container finished" podID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerID="c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7" exitCode=0
Nov 02 19:50:32.700888 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.700843    5224 generic.go:356] "Generic (PLEG): container finished" podID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerID="babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4" exitCode=0
Nov 02 19:50:32.700888 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.700854    5224 generic.go:356] "Generic (PLEG): container finished" podID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerID="298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d" exitCode=0
Nov 02 19:50:32.700888 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.700865    5224 generic.go:356] "Generic (PLEG): container finished" podID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerID="20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1" exitCode=0
Nov 02 19:50:32.700888 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.700873    5224 generic.go:356] "Generic (PLEG): container finished" podID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerID="5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156" exitCode=0
Nov 02 19:50:32.700888 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.700882    5224 generic.go:356] "Generic (PLEG): container finished" podID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerID="19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a" exitCode=0
Nov 02 19:50:32.701229 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.700881    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerDied","Data":"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"}
Nov 02 19:50:32.701229 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.700908    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.701229 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.700984    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerDied","Data":"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"}
Nov 02 19:50:32.701229 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.701013    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerDied","Data":"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"}
Nov 02 19:50:32.701229 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.701038    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerDied","Data":"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"}
Nov 02 19:50:32.701229 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.701054    5224 scope.go:122] "RemoveContainer" containerID="c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"
Nov 02 19:50:32.701229 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.701073    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerDied","Data":"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"}
Nov 02 19:50:32.701229 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.701097    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerDied","Data":"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"}
Nov 02 19:50:32.701229 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.701120    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2c5786c-518d-46f7-98c7-1d33f1b0c5d2","Type":"ContainerDied","Data":"0c46b4167cdc2f43c2b38900d940109ad3422093cb3b8752c3734ea0e388183b"}
Nov 02 19:50:32.709099 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.709077    5224 scope.go:122] "RemoveContainer" containerID="babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"
Nov 02 19:50:32.716548 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.716524    5224 scope.go:122] "RemoveContainer" containerID="298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"
Nov 02 19:50:32.723637 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.723614    5224 scope.go:122] "RemoveContainer" containerID="20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"
Nov 02 19:50:32.730722 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.730703    5224 scope.go:122] "RemoveContainer" containerID="5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"
Nov 02 19:50:32.737676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.737653    5224 scope.go:122] "RemoveContainer" containerID="19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"
Nov 02 19:50:32.745223 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.745200    5224 scope.go:122] "RemoveContainer" containerID="54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"
Nov 02 19:50:32.752086 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.752068    5224 scope.go:122] "RemoveContainer" containerID="c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"
Nov 02 19:50:32.752355 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:32.752335    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7\": container with ID starting with c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7 not found: ID does not exist" containerID="c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"
Nov 02 19:50:32.752399 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.752366    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"} err="failed to get container status \"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7\": rpc error: code = NotFound desc = could not find container \"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7\": container with ID starting with c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7 not found: ID does not exist"
Nov 02 19:50:32.752399 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.752387    5224 scope.go:122] "RemoveContainer" containerID="babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"
Nov 02 19:50:32.752627 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:32.752607    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4\": container with ID starting with babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4 not found: ID does not exist" containerID="babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"
Nov 02 19:50:32.752667 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.752637    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"} err="failed to get container status \"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4\": rpc error: code = NotFound desc = could not find container \"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4\": container with ID starting with babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4 not found: ID does not exist"
Nov 02 19:50:32.752667 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.752655    5224 scope.go:122] "RemoveContainer" containerID="298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"
Nov 02 19:50:32.752946 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:32.752929    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d\": container with ID starting with 298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d not found: ID does not exist" containerID="298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"
Nov 02 19:50:32.752983 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.752951    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"} err="failed to get container status \"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d\": rpc error: code = NotFound desc = could not find container \"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d\": container with ID starting with 298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d not found: ID does not exist"
Nov 02 19:50:32.752983 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.752965    5224 scope.go:122] "RemoveContainer" containerID="20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"
Nov 02 19:50:32.753178 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:32.753161    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1\": container with ID starting with 20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1 not found: ID does not exist" containerID="20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"
Nov 02 19:50:32.753217 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.753182    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"} err="failed to get container status \"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1\": rpc error: code = NotFound desc = could not find container \"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1\": container with ID starting with 20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1 not found: ID does not exist"
Nov 02 19:50:32.753217 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.753196    5224 scope.go:122] "RemoveContainer" containerID="5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"
Nov 02 19:50:32.753389 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:32.753373    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156\": container with ID starting with 5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156 not found: ID does not exist" containerID="5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"
Nov 02 19:50:32.753429 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.753392    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"} err="failed to get container status \"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156\": rpc error: code = NotFound desc = could not find container \"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156\": container with ID starting with 5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156 not found: ID does not exist"
Nov 02 19:50:32.753429 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.753404    5224 scope.go:122] "RemoveContainer" containerID="19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"
Nov 02 19:50:32.753600 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:32.753585    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a\": container with ID starting with 19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a not found: ID does not exist" containerID="19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"
Nov 02 19:50:32.753638 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.753603    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"} err="failed to get container status \"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a\": rpc error: code = NotFound desc = could not find container \"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a\": container with ID starting with 19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a not found: ID does not exist"
Nov 02 19:50:32.753638 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.753615    5224 scope.go:122] "RemoveContainer" containerID="54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"
Nov 02 19:50:32.753882 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:32.753864    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038\": container with ID starting with 54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038 not found: ID does not exist" containerID="54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"
Nov 02 19:50:32.753927 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.753887    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"} err="failed to get container status \"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038\": rpc error: code = NotFound desc = could not find container \"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038\": container with ID starting with 54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038 not found: ID does not exist"
Nov 02 19:50:32.753927 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.753899    5224 scope.go:122] "RemoveContainer" containerID="c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"
Nov 02 19:50:32.754123 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.754098    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"} err="failed to get container status \"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7\": rpc error: code = NotFound desc = could not find container \"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7\": container with ID starting with c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7 not found: ID does not exist"
Nov 02 19:50:32.754199 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.754125    5224 scope.go:122] "RemoveContainer" containerID="babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"
Nov 02 19:50:32.754410 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.754393    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"} err="failed to get container status \"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4\": rpc error: code = NotFound desc = could not find container \"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4\": container with ID starting with babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4 not found: ID does not exist"
Nov 02 19:50:32.754410 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.754408    5224 scope.go:122] "RemoveContainer" containerID="298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"
Nov 02 19:50:32.754583 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.754565    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"} err="failed to get container status \"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d\": rpc error: code = NotFound desc = could not find container \"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d\": container with ID starting with 298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d not found: ID does not exist"
Nov 02 19:50:32.754656 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.754585    5224 scope.go:122] "RemoveContainer" containerID="20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"
Nov 02 19:50:32.754793 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.754755    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"} err="failed to get container status \"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1\": rpc error: code = NotFound desc = could not find container \"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1\": container with ID starting with 20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1 not found: ID does not exist"
Nov 02 19:50:32.754855 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.754795    5224 scope.go:122] "RemoveContainer" containerID="5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"
Nov 02 19:50:32.755092 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.755072    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"} err="failed to get container status \"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156\": rpc error: code = NotFound desc = could not find container \"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156\": container with ID starting with 5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156 not found: ID does not exist"
Nov 02 19:50:32.755158 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.755093    5224 scope.go:122] "RemoveContainer" containerID="19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"
Nov 02 19:50:32.755330 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.755312    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"} err="failed to get container status \"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a\": rpc error: code = NotFound desc = could not find container \"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a\": container with ID starting with 19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a not found: ID does not exist"
Nov 02 19:50:32.755399 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.755330    5224 scope.go:122] "RemoveContainer" containerID="54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"
Nov 02 19:50:32.755573 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.755551    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"} err="failed to get container status \"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038\": rpc error: code = NotFound desc = could not find container \"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038\": container with ID starting with 54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038 not found: ID does not exist"
Nov 02 19:50:32.755632 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.755573    5224 scope.go:122] "RemoveContainer" containerID="c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"
Nov 02 19:50:32.755819 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.755797    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"} err="failed to get container status \"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7\": rpc error: code = NotFound desc = could not find container \"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7\": container with ID starting with c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7 not found: ID does not exist"
Nov 02 19:50:32.755819 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.755818    5224 scope.go:122] "RemoveContainer" containerID="babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"
Nov 02 19:50:32.756041 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.756025    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"} err="failed to get container status \"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4\": rpc error: code = NotFound desc = could not find container \"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4\": container with ID starting with babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4 not found: ID does not exist"
Nov 02 19:50:32.756041 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.756040    5224 scope.go:122] "RemoveContainer" containerID="298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"
Nov 02 19:50:32.756280 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.756262    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"} err="failed to get container status \"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d\": rpc error: code = NotFound desc = could not find container \"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d\": container with ID starting with 298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d not found: ID does not exist"
Nov 02 19:50:32.756340 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.756280    5224 scope.go:122] "RemoveContainer" containerID="20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"
Nov 02 19:50:32.756509 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.756489    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"} err="failed to get container status \"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1\": rpc error: code = NotFound desc = could not find container \"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1\": container with ID starting with 20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1 not found: ID does not exist"
Nov 02 19:50:32.756579 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.756509    5224 scope.go:122] "RemoveContainer" containerID="5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"
Nov 02 19:50:32.756727 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.756708    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"} err="failed to get container status \"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156\": rpc error: code = NotFound desc = could not find container \"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156\": container with ID starting with 5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156 not found: ID does not exist"
Nov 02 19:50:32.756814 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.756727    5224 scope.go:122] "RemoveContainer" containerID="19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"
Nov 02 19:50:32.756964 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.756941    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"} err="failed to get container status \"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a\": rpc error: code = NotFound desc = could not find container \"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a\": container with ID starting with 19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a not found: ID does not exist"
Nov 02 19:50:32.757030 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.756964    5224 scope.go:122] "RemoveContainer" containerID="54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"
Nov 02 19:50:32.757183 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.757163    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"} err="failed to get container status \"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038\": rpc error: code = NotFound desc = could not find container \"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038\": container with ID starting with 54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038 not found: ID does not exist"
Nov 02 19:50:32.757257 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.757184    5224 scope.go:122] "RemoveContainer" containerID="c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"
Nov 02 19:50:32.757447 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.757429    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"} err="failed to get container status \"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7\": rpc error: code = NotFound desc = could not find container \"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7\": container with ID starting with c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7 not found: ID does not exist"
Nov 02 19:50:32.757486 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.757447    5224 scope.go:122] "RemoveContainer" containerID="babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"
Nov 02 19:50:32.757660 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.757642    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"} err="failed to get container status \"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4\": rpc error: code = NotFound desc = could not find container \"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4\": container with ID starting with babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4 not found: ID does not exist"
Nov 02 19:50:32.757720 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.757661    5224 scope.go:122] "RemoveContainer" containerID="298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"
Nov 02 19:50:32.757908 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.757889    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"} err="failed to get container status \"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d\": rpc error: code = NotFound desc = could not find container \"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d\": container with ID starting with 298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d not found: ID does not exist"
Nov 02 19:50:32.757963 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.757911    5224 scope.go:122] "RemoveContainer" containerID="20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"
Nov 02 19:50:32.758117 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.758103    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"} err="failed to get container status \"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1\": rpc error: code = NotFound desc = could not find container \"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1\": container with ID starting with 20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1 not found: ID does not exist"
Nov 02 19:50:32.758156 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.758117    5224 scope.go:122] "RemoveContainer" containerID="5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"
Nov 02 19:50:32.758286 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.758272    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"} err="failed to get container status \"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156\": rpc error: code = NotFound desc = could not find container \"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156\": container with ID starting with 5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156 not found: ID does not exist"
Nov 02 19:50:32.758286 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.758285    5224 scope.go:122] "RemoveContainer" containerID="19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"
Nov 02 19:50:32.758491 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.758470    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"} err="failed to get container status \"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a\": rpc error: code = NotFound desc = could not find container \"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a\": container with ID starting with 19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a not found: ID does not exist"
Nov 02 19:50:32.758534 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.758492    5224 scope.go:122] "RemoveContainer" containerID="54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"
Nov 02 19:50:32.758755 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.758735    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"} err="failed to get container status \"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038\": rpc error: code = NotFound desc = could not find container \"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038\": container with ID starting with 54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038 not found: ID does not exist"
Nov 02 19:50:32.758807 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.758755    5224 scope.go:122] "RemoveContainer" containerID="c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"
Nov 02 19:50:32.758983 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.758967    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"} err="failed to get container status \"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7\": rpc error: code = NotFound desc = could not find container \"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7\": container with ID starting with c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7 not found: ID does not exist"
Nov 02 19:50:32.759036 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.758984    5224 scope.go:122] "RemoveContainer" containerID="babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"
Nov 02 19:50:32.759204 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.759184    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"} err="failed to get container status \"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4\": rpc error: code = NotFound desc = could not find container \"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4\": container with ID starting with babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4 not found: ID does not exist"
Nov 02 19:50:32.759204 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.759202    5224 scope.go:122] "RemoveContainer" containerID="298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"
Nov 02 19:50:32.759407 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.759389    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"} err="failed to get container status \"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d\": rpc error: code = NotFound desc = could not find container \"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d\": container with ID starting with 298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d not found: ID does not exist"
Nov 02 19:50:32.759445 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.759408    5224 scope.go:122] "RemoveContainer" containerID="20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"
Nov 02 19:50:32.759601 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.759584    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"} err="failed to get container status \"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1\": rpc error: code = NotFound desc = could not find container \"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1\": container with ID starting with 20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1 not found: ID does not exist"
Nov 02 19:50:32.759642 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.759601    5224 scope.go:122] "RemoveContainer" containerID="5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"
Nov 02 19:50:32.759834 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.759816    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"} err="failed to get container status \"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156\": rpc error: code = NotFound desc = could not find container \"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156\": container with ID starting with 5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156 not found: ID does not exist"
Nov 02 19:50:32.759875 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.759835    5224 scope.go:122] "RemoveContainer" containerID="19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"
Nov 02 19:50:32.760040 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.760023    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"} err="failed to get container status \"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a\": rpc error: code = NotFound desc = could not find container \"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a\": container with ID starting with 19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a not found: ID does not exist"
Nov 02 19:50:32.760081 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.760040    5224 scope.go:122] "RemoveContainer" containerID="54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"
Nov 02 19:50:32.760241 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.760221    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"} err="failed to get container status \"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038\": rpc error: code = NotFound desc = could not find container \"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038\": container with ID starting with 54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038 not found: ID does not exist"
Nov 02 19:50:32.760285 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.760242    5224 scope.go:122] "RemoveContainer" containerID="c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"
Nov 02 19:50:32.760421 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.760402    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7"} err="failed to get container status \"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7\": rpc error: code = NotFound desc = could not find container \"c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7\": container with ID starting with c4cdb087b1795dc1691485e7b7c00639b116f85f3287aef715a72e63278b52f7 not found: ID does not exist"
Nov 02 19:50:32.760459 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.760421    5224 scope.go:122] "RemoveContainer" containerID="babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"
Nov 02 19:50:32.760604 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.760588    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4"} err="failed to get container status \"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4\": rpc error: code = NotFound desc = could not find container \"babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4\": container with ID starting with babd36e77281bf95efc0ba6e8f4a1d99a77c31bb53c9860a2a43b142c2f829e4 not found: ID does not exist"
Nov 02 19:50:32.760646 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.760604    5224 scope.go:122] "RemoveContainer" containerID="298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"
Nov 02 19:50:32.760799 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.760783    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d"} err="failed to get container status \"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d\": rpc error: code = NotFound desc = could not find container \"298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d\": container with ID starting with 298dad4bc0a796b6570985054c7ef7cd5e5b7f3c7522affffc1bed3942298b7d not found: ID does not exist"
Nov 02 19:50:32.760847 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.760798    5224 scope.go:122] "RemoveContainer" containerID="20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"
Nov 02 19:50:32.760979 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.760960    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1"} err="failed to get container status \"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1\": rpc error: code = NotFound desc = could not find container \"20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1\": container with ID starting with 20224aaaf0e3a0c6f8288482edfb2530c130055c64b303884d9c052e22257ee1 not found: ID does not exist"
Nov 02 19:50:32.761017 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.760980    5224 scope.go:122] "RemoveContainer" containerID="5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"
Nov 02 19:50:32.761201 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.761186    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156"} err="failed to get container status \"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156\": rpc error: code = NotFound desc = could not find container \"5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156\": container with ID starting with 5c83f4e3902fc30a32110f0578dd069b02ccc8893173dd46bb2e9ff1beb03156 not found: ID does not exist"
Nov 02 19:50:32.761238 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.761201    5224 scope.go:122] "RemoveContainer" containerID="19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"
Nov 02 19:50:32.761413 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.761388    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a"} err="failed to get container status \"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a\": rpc error: code = NotFound desc = could not find container \"19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a\": container with ID starting with 19e641fff5e1522d47d2ad2b56052a02e7e3353419cf557c9a4ede9fbf16244a not found: ID does not exist"
Nov 02 19:50:32.761451 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.761414    5224 scope.go:122] "RemoveContainer" containerID="54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"
Nov 02 19:50:32.761610 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.761595    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038"} err="failed to get container status \"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038\": rpc error: code = NotFound desc = could not find container \"54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038\": container with ID starting with 54e2fca8295b8bcaed97485be2fc7141976efd97c9c4fab31a735d0f4ea98038 not found: ID does not exist"
Nov 02 19:50:32.871757 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.871723    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:50:32.875737 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.875707    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:50:32.910946 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.910910    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:50:32.911231 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911216    5224 cpu_manager.go:414] "containerMap: removing container" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="thanos-sidecar"
Nov 02 19:50:32.911274 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911233    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="thanos-sidecar"
Nov 02 19:50:32.911274 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911247    5224 cpu_manager.go:414] "containerMap: removing container" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="kube-rbac-proxy-web"
Nov 02 19:50:32.911274 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911252    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="kube-rbac-proxy-web"
Nov 02 19:50:32.911274 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911259    5224 cpu_manager.go:414] "containerMap: removing container" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="kube-rbac-proxy-thanos"
Nov 02 19:50:32.911274 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911264    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="kube-rbac-proxy-thanos"
Nov 02 19:50:32.911411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911282    5224 cpu_manager.go:414] "containerMap: removing container" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="config-reloader"
Nov 02 19:50:32.911411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911287    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="config-reloader"
Nov 02 19:50:32.911411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911295    5224 cpu_manager.go:414] "containerMap: removing container" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="kube-rbac-proxy"
Nov 02 19:50:32.911411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911300    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="kube-rbac-proxy"
Nov 02 19:50:32.911411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911313    5224 cpu_manager.go:414] "containerMap: removing container" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="prometheus"
Nov 02 19:50:32.911411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911318    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="prometheus"
Nov 02 19:50:32.911411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911324    5224 cpu_manager.go:414] "containerMap: removing container" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="init-config-reloader"
Nov 02 19:50:32.911411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911329    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="init-config-reloader"
Nov 02 19:50:32.911411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911371    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="config-reloader"
Nov 02 19:50:32.911411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911379    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="kube-rbac-proxy"
Nov 02 19:50:32.911411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911387    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="prometheus"
Nov 02 19:50:32.911411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911393    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="thanos-sidecar"
Nov 02 19:50:32.911722 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911419    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="kube-rbac-proxy-web"
Nov 02 19:50:32.911722 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.911424    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" containerName="kube-rbac-proxy-thanos"
Nov 02 19:50:32.915405 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.915381    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.969974 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.969871    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 19:50:32.969974 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.969871    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 19:50:32.969974 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.969912    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 19:50:32.969974 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.969872    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 19:50:32.969974 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.969871    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 19:50:32.970439 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.970129    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 19:50:32.970439 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.970133    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 19:50:32.970439 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.970173    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 19:50:32.970439 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.970250    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 19:50:32.970439 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.970285    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 19:50:32.970439 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.970294    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a6pl4uph7s6pt\""
Nov 02 19:50:32.970439 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.970338    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mvxpm\""
Nov 02 19:50:32.970439 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.970368    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 19:50:32.970439 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.970404    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 19:50:32.970439 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.970441    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 19:50:32.982182 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982149    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd453033-2746-4b42-aabc-c364430b8ce3-config-out\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982182 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982185    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982215    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982368 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982336    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982443 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982367    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982443 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982395    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-config\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982532 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982459    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982586 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982569    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982640 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982610    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982693 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982640    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd453033-2746-4b42-aabc-c364430b8ce3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982789 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982754    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcl6g\" (UniqueName: \"kubernetes.io/projected/fd453033-2746-4b42-aabc-c364430b8ce3-kube-api-access-xcl6g\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982829 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982802    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982871 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982841    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982914 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982874    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.982914 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982892    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.983020 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.982989    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.983052 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.983010    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.983052 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.983037    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.983128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.983067    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-web-config\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:32.983128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:32.983107    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.022224 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.022178    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 19:50:33.022510 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.022490    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 19:50:33.027882 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.027851    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:50:33.083977 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.083937    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.083977 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.083976    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.084221 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.084001    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-web-config\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.084221 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.084021    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.084325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.084222    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd453033-2746-4b42-aabc-c364430b8ce3-config-out\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.084325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.084268    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.084325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.084310    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.084467 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.084341    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.084467 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.084371    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.084893 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.084861    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.085419 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.085391    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-config\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.085533 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.085444    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.085533 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.085499    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.085643 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.085543    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.085643 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.085577    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd453033-2746-4b42-aabc-c364430b8ce3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.085643 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.085620    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcl6g\" (UniqueName: \"kubernetes.io/projected/fd453033-2746-4b42-aabc-c364430b8ce3-kube-api-access-xcl6g\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.085805 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.085659    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.085805 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.085698    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.085805 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.085723    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.085805 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.085752    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.085996 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.085815    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.087441 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.086603    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.087441 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.087102    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.087441 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.087383    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.087644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.087603    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.087968 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.087790    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.087968 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.087811    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.087968 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.087921    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd453033-2746-4b42-aabc-c364430b8ce3-config-out\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.088174 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.088013    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-web-config\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.088174 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.088087    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.088275 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.088217    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.088275 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.088259    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.088962 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.088934    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd453033-2746-4b42-aabc-c364430b8ce3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.089061 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.089007    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.089418 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.089393    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.089535 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.089521    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.089917 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.089898    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-config\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.090544 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.090529    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd453033-2746-4b42-aabc-c364430b8ce3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.090625 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.090606    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd453033-2746-4b42-aabc-c364430b8ce3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.145879 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.145848    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcl6g\" (UniqueName: \"kubernetes.io/projected/fd453033-2746-4b42-aabc-c364430b8ce3-kube-api-access-xcl6g\") pod \"prometheus-k8s-0\" (UID: \"fd453033-2746-4b42-aabc-c364430b8ce3\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.166572 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.166529    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c5786c-518d-46f7-98c7-1d33f1b0c5d2" path="/var/lib/kubelet/pods/f2c5786c-518d-46f7-98c7-1d33f1b0c5d2/volumes"
Nov 02 19:50:33.232296 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.232206    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:33.652585 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:50:33.652541    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd453033_2746_4b42_aabc_c364430b8ce3.slice/crio-bc6e0a5b3e3f2064b4f43697d66047f9993175999f3937e3805c4dc3af5dda2d WatchSource:0}: Error finding container bc6e0a5b3e3f2064b4f43697d66047f9993175999f3937e3805c4dc3af5dda2d: Status 404 returned error can't find the container with id bc6e0a5b3e3f2064b4f43697d66047f9993175999f3937e3805c4dc3af5dda2d
Nov 02 19:50:33.655625 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.655594    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 19:50:33.707135 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:33.707106    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd453033-2746-4b42-aabc-c364430b8ce3","Type":"ContainerStarted","Data":"bc6e0a5b3e3f2064b4f43697d66047f9993175999f3937e3805c4dc3af5dda2d"}
Nov 02 19:50:34.712915 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:34.712879    5224 generic.go:356] "Generic (PLEG): container finished" podID="fd453033-2746-4b42-aabc-c364430b8ce3" containerID="f4582056ecedf639da28cd6d23d481d0c61a5074a13916682589f11c524efe4a" exitCode=0
Nov 02 19:50:34.713330 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:34.712951    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd453033-2746-4b42-aabc-c364430b8ce3","Type":"ContainerDied","Data":"f4582056ecedf639da28cd6d23d481d0c61a5074a13916682589f11c524efe4a"}
Nov 02 19:50:35.720643 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:35.720604    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd453033-2746-4b42-aabc-c364430b8ce3","Type":"ContainerStarted","Data":"27a7766c02eab7d86347a9c44fc6cd18de2ec9f5da153ee3a8fbcd982ef49a43"}
Nov 02 19:50:35.721098 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:35.720676    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd453033-2746-4b42-aabc-c364430b8ce3","Type":"ContainerStarted","Data":"5eec5b6d3656a22bb6879d5d357bf035860e9cb887a3a2940111eceeb362857c"}
Nov 02 19:50:35.721098 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:35.720704    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd453033-2746-4b42-aabc-c364430b8ce3","Type":"ContainerStarted","Data":"f6e1af4da9f36830446c20f8aaa8cd9a20c65a4116a05bbd7132775b834eb02e"}
Nov 02 19:50:35.721098 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:35.720734    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd453033-2746-4b42-aabc-c364430b8ce3","Type":"ContainerStarted","Data":"3d01f20d59b73a33153bd05959431dd77669096382588bf7a8539424452b5188"}
Nov 02 19:50:35.721098 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:35.720757    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd453033-2746-4b42-aabc-c364430b8ce3","Type":"ContainerStarted","Data":"5547079454db4c6ead61a66b3d795b3084cca4a9dc2ed9ff16c1821126ae6c32"}
Nov 02 19:50:35.721098 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:35.720810    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd453033-2746-4b42-aabc-c364430b8ce3","Type":"ContainerStarted","Data":"dd6f1dda4ca7f2f1bec0f87a7c45352991c5fedb2cf8c2905fc90ab03f5f6c6d"}
Nov 02 19:50:35.847613 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:35.847561    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.8304722509999998 podStartE2EDuration="3.847546252s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.074001ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:50:35.847364741 +0000 UTC m=+116.845053350" watchObservedRunningTime="2026-11-02 19:50:35.847546252 +0000 UTC m=+116.845234855"
Nov 02 19:50:38.233044 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:38.232980    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:50:45.313452 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:45.313408    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:50:46.433269 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:50:46.433226    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:50:57.357465 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:50:57.357428    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:51:27.357105 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:51:27.357064    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:51:33.232750 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:51:33.232700    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:51:33.248056 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:51:33.248031    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:51:33.945352 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:51:33.945326    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 19:51:46.662434 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:51:46.662392    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:51:46.880070 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:51:46.880033    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:51:57.357191 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:51:57.357113    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:52:25.456134 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:25.456100    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:52:27.358152 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:52:27.358115    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:52:27.370685 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.370658    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"]
Nov 02 19:52:27.372959 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.372943    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"
Nov 02 19:52:27.427492 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.427462    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"]
Nov 02 19:52:27.478187 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.478160    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 19:52:27.478187 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.478185    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 19:52:27.478382 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.478213    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-2hf7r\""
Nov 02 19:52:27.480986 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.480969    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"]
Nov 02 19:52:27.483356 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.483342    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"
Nov 02 19:52:27.501585 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.501559    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p\" (UID: \"d2ca3136-91c1-499c-a96f-ea1328393a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"
Nov 02 19:52:27.501712 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.501612    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p\" (UID: \"d2ca3136-91c1-499c-a96f-ea1328393a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"
Nov 02 19:52:27.501797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.501736    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6xc4\" (UniqueName: \"kubernetes.io/projected/d2ca3136-91c1-499c-a96f-ea1328393a0b-kube-api-access-b6xc4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p\" (UID: \"d2ca3136-91c1-499c-a96f-ea1328393a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"
Nov 02 19:52:27.534580 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.534549    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"]
Nov 02 19:52:27.536695 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.536674    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-ctw8r\""
Nov 02 19:52:27.574124 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.574090    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"]
Nov 02 19:52:27.576830 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.576810    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"
Nov 02 19:52:27.602641 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.602619    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p\" (UID: \"d2ca3136-91c1-499c-a96f-ea1328393a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"
Nov 02 19:52:27.602749 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.602653    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w\" (UID: \"29c18c81-02ef-422b-948c-7a9859862c2e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"
Nov 02 19:52:27.602749 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.602671    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w\" (UID: \"29c18c81-02ef-422b-948c-7a9859862c2e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"
Nov 02 19:52:27.602841 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.602747    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6xc4\" (UniqueName: \"kubernetes.io/projected/d2ca3136-91c1-499c-a96f-ea1328393a0b-kube-api-access-b6xc4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p\" (UID: \"d2ca3136-91c1-499c-a96f-ea1328393a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"
Nov 02 19:52:27.602841 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.602821    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tp6c\" (UniqueName: \"kubernetes.io/projected/29c18c81-02ef-422b-948c-7a9859862c2e-kube-api-access-7tp6c\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w\" (UID: \"29c18c81-02ef-422b-948c-7a9859862c2e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"
Nov 02 19:52:27.602904 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.602859    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p\" (UID: \"d2ca3136-91c1-499c-a96f-ea1328393a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"
Nov 02 19:52:27.603153 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.603136    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p\" (UID: \"d2ca3136-91c1-499c-a96f-ea1328393a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"
Nov 02 19:52:27.603189 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.603174    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p\" (UID: \"d2ca3136-91c1-499c-a96f-ea1328393a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"
Nov 02 19:52:27.630421 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.630362    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"]
Nov 02 19:52:27.636911 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.636894    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-9vwqk\""
Nov 02 19:52:27.662943 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.662907    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6xc4\" (UniqueName: \"kubernetes.io/projected/d2ca3136-91c1-499c-a96f-ea1328393a0b-kube-api-access-b6xc4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p\" (UID: \"d2ca3136-91c1-499c-a96f-ea1328393a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"
Nov 02 19:52:27.669726 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.669702    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"]
Nov 02 19:52:27.672177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.672160    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"
Nov 02 19:52:27.672475 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.672455    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"]
Nov 02 19:52:27.674498 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.674480    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"
Nov 02 19:52:27.684824 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.684702    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"
Nov 02 19:52:27.704293 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.704270    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tp6c\" (UniqueName: \"kubernetes.io/projected/29c18c81-02ef-422b-948c-7a9859862c2e-kube-api-access-7tp6c\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w\" (UID: \"29c18c81-02ef-422b-948c-7a9859862c2e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"
Nov 02 19:52:27.704396 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.704322    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk\" (UID: \"62465b5f-e6e2-4734-8897-1c314c6429af\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"
Nov 02 19:52:27.704396 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.704369    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxx4\" (UniqueName: \"kubernetes.io/projected/62465b5f-e6e2-4734-8897-1c314c6429af-kube-api-access-6zxx4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk\" (UID: \"62465b5f-e6e2-4734-8897-1c314c6429af\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"
Nov 02 19:52:27.704508 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.704489    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w\" (UID: \"29c18c81-02ef-422b-948c-7a9859862c2e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"
Nov 02 19:52:27.704561 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.704520    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w\" (UID: \"29c18c81-02ef-422b-948c-7a9859862c2e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"
Nov 02 19:52:27.704561 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.704541    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk\" (UID: \"62465b5f-e6e2-4734-8897-1c314c6429af\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"
Nov 02 19:52:27.704862 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.704837    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w\" (UID: \"29c18c81-02ef-422b-948c-7a9859862c2e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"
Nov 02 19:52:27.704918 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.704869    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w\" (UID: \"29c18c81-02ef-422b-948c-7a9859862c2e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"
Nov 02 19:52:27.727278 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.727248    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"]
Nov 02 19:52:27.733138 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.733111    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-dxlc4\""
Nov 02 19:52:27.735174 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.734424    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"]
Nov 02 19:52:27.737353 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.737333    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-nq2gj\""
Nov 02 19:52:27.761715 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.761665    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:52:27.763967 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.763937    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tp6c\" (UniqueName: \"kubernetes.io/projected/29c18c81-02ef-422b-948c-7a9859862c2e-kube-api-access-7tp6c\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w\" (UID: \"29c18c81-02ef-422b-948c-7a9859862c2e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"
Nov 02 19:52:27.795575 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.795549    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"
Nov 02 19:52:27.804964 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.804931    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk\" (UID: \"62465b5f-e6e2-4734-8897-1c314c6429af\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"
Nov 02 19:52:27.805060 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.804989    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh\" (UID: \"c1097c45-59cb-4dc8-a416-989e95019e9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"
Nov 02 19:52:27.805060 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.805026    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxx4\" (UniqueName: \"kubernetes.io/projected/62465b5f-e6e2-4734-8897-1c314c6429af-kube-api-access-6zxx4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk\" (UID: \"62465b5f-e6e2-4734-8897-1c314c6429af\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"
Nov 02 19:52:27.805164 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.805129    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n\" (UID: \"9d018247-7db6-42ca-a57b-b53dc04c0e67\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"
Nov 02 19:52:27.805209 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.805187    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n\" (UID: \"9d018247-7db6-42ca-a57b-b53dc04c0e67\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"
Nov 02 19:52:27.805255 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.805217    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh\" (UID: \"c1097c45-59cb-4dc8-a416-989e95019e9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"
Nov 02 19:52:27.805255 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.805249    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk\" (UID: \"62465b5f-e6e2-4734-8897-1c314c6429af\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"
Nov 02 19:52:27.805388 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.805372    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4cm4\" (UniqueName: \"kubernetes.io/projected/c1097c45-59cb-4dc8-a416-989e95019e9b-kube-api-access-t4cm4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh\" (UID: \"c1097c45-59cb-4dc8-a416-989e95019e9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"
Nov 02 19:52:27.805455 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.805388    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk\" (UID: \"62465b5f-e6e2-4734-8897-1c314c6429af\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"
Nov 02 19:52:27.805455 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.805421    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv4qh\" (UniqueName: \"kubernetes.io/projected/9d018247-7db6-42ca-a57b-b53dc04c0e67-kube-api-access-xv4qh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n\" (UID: \"9d018247-7db6-42ca-a57b-b53dc04c0e67\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"
Nov 02 19:52:27.805590 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.805572    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk\" (UID: \"62465b5f-e6e2-4734-8897-1c314c6429af\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"
Nov 02 19:52:27.863673 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.863644    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxx4\" (UniqueName: \"kubernetes.io/projected/62465b5f-e6e2-4734-8897-1c314c6429af-kube-api-access-6zxx4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk\" (UID: \"62465b5f-e6e2-4734-8897-1c314c6429af\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"
Nov 02 19:52:27.889714 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.889648    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"
Nov 02 19:52:27.905965 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.905929    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n\" (UID: \"9d018247-7db6-42ca-a57b-b53dc04c0e67\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"
Nov 02 19:52:27.906115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.905977    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n\" (UID: \"9d018247-7db6-42ca-a57b-b53dc04c0e67\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"
Nov 02 19:52:27.906115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.906009    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh\" (UID: \"c1097c45-59cb-4dc8-a416-989e95019e9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"
Nov 02 19:52:27.906115 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.906065    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4cm4\" (UniqueName: \"kubernetes.io/projected/c1097c45-59cb-4dc8-a416-989e95019e9b-kube-api-access-t4cm4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh\" (UID: \"c1097c45-59cb-4dc8-a416-989e95019e9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"
Nov 02 19:52:27.906295 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.906120    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv4qh\" (UniqueName: \"kubernetes.io/projected/9d018247-7db6-42ca-a57b-b53dc04c0e67-kube-api-access-xv4qh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n\" (UID: \"9d018247-7db6-42ca-a57b-b53dc04c0e67\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"
Nov 02 19:52:27.906295 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.906164    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh\" (UID: \"c1097c45-59cb-4dc8-a416-989e95019e9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"
Nov 02 19:52:27.906295 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.906271    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n\" (UID: \"9d018247-7db6-42ca-a57b-b53dc04c0e67\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"
Nov 02 19:52:27.906460 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.906330    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n\" (UID: \"9d018247-7db6-42ca-a57b-b53dc04c0e67\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"
Nov 02 19:52:27.906609 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.906589    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh\" (UID: \"c1097c45-59cb-4dc8-a416-989e95019e9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"
Nov 02 19:52:27.906816 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.906795    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh\" (UID: \"c1097c45-59cb-4dc8-a416-989e95019e9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"
Nov 02 19:52:27.964750 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.964720    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv4qh\" (UniqueName: \"kubernetes.io/projected/9d018247-7db6-42ca-a57b-b53dc04c0e67-kube-api-access-xv4qh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n\" (UID: \"9d018247-7db6-42ca-a57b-b53dc04c0e67\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"
Nov 02 19:52:27.964906 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.964732    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4cm4\" (UniqueName: \"kubernetes.io/projected/c1097c45-59cb-4dc8-a416-989e95019e9b-kube-api-access-t4cm4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh\" (UID: \"c1097c45-59cb-4dc8-a416-989e95019e9b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"
Nov 02 19:52:27.985162 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.985138    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"
Nov 02 19:52:27.990668 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:27.990644    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"
Nov 02 19:52:28.051094 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:28.050966    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"]
Nov 02 19:52:28.054482 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:52:28.054429    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ca3136_91c1_499c_a96f_ea1328393a0b.slice/crio-75ed0088bd1d553e5494a163fe98a032574ebcc159e96dcf05b6e1850ff809df WatchSource:0}: Error finding container 75ed0088bd1d553e5494a163fe98a032574ebcc159e96dcf05b6e1850ff809df: Status 404 returned error can't find the container with id 75ed0088bd1d553e5494a163fe98a032574ebcc159e96dcf05b6e1850ff809df
Nov 02 19:52:28.112080 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:28.112048    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p" event={"ID":"d2ca3136-91c1-499c-a96f-ea1328393a0b","Type":"ContainerStarted","Data":"75ed0088bd1d553e5494a163fe98a032574ebcc159e96dcf05b6e1850ff809df"}
Nov 02 19:52:28.160671 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:52:28.160645    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c18c81_02ef_422b_948c_7a9859862c2e.slice/crio-4b789a3266692ff4c6ba4dbbdc4b8ec6f9ca1405ba0ae625259777d515cc1c13 WatchSource:0}: Error finding container 4b789a3266692ff4c6ba4dbbdc4b8ec6f9ca1405ba0ae625259777d515cc1c13: Status 404 returned error can't find the container with id 4b789a3266692ff4c6ba4dbbdc4b8ec6f9ca1405ba0ae625259777d515cc1c13
Nov 02 19:52:28.162693 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:28.162670    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"]
Nov 02 19:52:28.253420 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:52:28.253390    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62465b5f_e6e2_4734_8897_1c314c6429af.slice/crio-da95a5a732b91158f4752dacdc0ea030ed608e6ddcf47d6951766592a1029d59 WatchSource:0}: Error finding container da95a5a732b91158f4752dacdc0ea030ed608e6ddcf47d6951766592a1029d59: Status 404 returned error can't find the container with id da95a5a732b91158f4752dacdc0ea030ed608e6ddcf47d6951766592a1029d59
Nov 02 19:52:28.255432 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:28.255412    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"]
Nov 02 19:52:28.351823 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:52:28.351793    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1097c45_59cb_4dc8_a416_989e95019e9b.slice/crio-a87dcd794ead9f49e47c72320589daf7a248f391f40e667ed5ad8cbb470f5d2c WatchSource:0}: Error finding container a87dcd794ead9f49e47c72320589daf7a248f391f40e667ed5ad8cbb470f5d2c: Status 404 returned error can't find the container with id a87dcd794ead9f49e47c72320589daf7a248f391f40e667ed5ad8cbb470f5d2c
Nov 02 19:52:28.353805 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:28.353785    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"]
Nov 02 19:52:28.360275 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:52:28.360251    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d018247_7db6_42ca_a57b_b53dc04c0e67.slice/crio-fa67494ca8ddbdbf88aada084c4abad3fb9e8f6d47366a8b384f7802a7147dd6 WatchSource:0}: Error finding container fa67494ca8ddbdbf88aada084c4abad3fb9e8f6d47366a8b384f7802a7147dd6: Status 404 returned error can't find the container with id fa67494ca8ddbdbf88aada084c4abad3fb9e8f6d47366a8b384f7802a7147dd6
Nov 02 19:52:28.366683 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:28.362575    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"]
Nov 02 19:52:29.117744 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:29.117689    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n" event={"ID":"9d018247-7db6-42ca-a57b-b53dc04c0e67","Type":"ContainerStarted","Data":"fa67494ca8ddbdbf88aada084c4abad3fb9e8f6d47366a8b384f7802a7147dd6"}
Nov 02 19:52:29.119515 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:29.119480    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh" event={"ID":"c1097c45-59cb-4dc8-a416-989e95019e9b","Type":"ContainerStarted","Data":"a87dcd794ead9f49e47c72320589daf7a248f391f40e667ed5ad8cbb470f5d2c"}
Nov 02 19:52:29.121321 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:29.121279    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk" event={"ID":"62465b5f-e6e2-4734-8897-1c314c6429af","Type":"ContainerStarted","Data":"da95a5a732b91158f4752dacdc0ea030ed608e6ddcf47d6951766592a1029d59"}
Nov 02 19:52:29.122679 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:29.122647    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w" event={"ID":"29c18c81-02ef-422b-948c-7a9859862c2e","Type":"ContainerStarted","Data":"4b789a3266692ff4c6ba4dbbdc4b8ec6f9ca1405ba0ae625259777d515cc1c13"}
Nov 02 19:52:33.309931 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:33.309888    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:52:33.535259 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:33.535210    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:52:34.433669 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.433567    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:52:34.609790 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.609421    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"]
Nov 02 19:52:34.614732 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.613882    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.669028 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.668952    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"]
Nov 02 19:52:34.673782 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.673740    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.673943 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.673810    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.673943 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.673853    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.673943 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.673873    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.673943 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.673898    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e02684d-c911-4625-ade1-6b04120058d8-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.673943 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.673917    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.673943 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.673939    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e02684d-c911-4625-ade1-6b04120058d8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.674225 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.673993    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e02684d-c911-4625-ade1-6b04120058d8-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.674225 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.674057    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzmnw\" (UniqueName: \"kubernetes.io/projected/3e02684d-c911-4625-ade1-6b04120058d8-kube-api-access-gzmnw\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.707269 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.707238    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:52:34.709051 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.709030    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 19:52:34.709441 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.709422    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 19:52:34.709546 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.709502    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 19:52:34.718938 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.718902    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-c5g28\""
Nov 02 19:52:34.776869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.775017    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzmnw\" (UniqueName: \"kubernetes.io/projected/3e02684d-c911-4625-ade1-6b04120058d8-kube-api-access-gzmnw\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.776869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.775089    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.776869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.775123    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.776869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.775160    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.776869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.775183    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.776869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.775211    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e02684d-c911-4625-ade1-6b04120058d8-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.776869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.775242    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.776869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.775278    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e02684d-c911-4625-ade1-6b04120058d8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.776869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.775332    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e02684d-c911-4625-ade1-6b04120058d8-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.776869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.776048    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.776869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.776275    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.776869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.776505    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.776869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.776527    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.777671 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.777149    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e02684d-c911-4625-ade1-6b04120058d8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.780410 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.780385    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e02684d-c911-4625-ade1-6b04120058d8-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.780890 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.780868    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e02684d-c911-4625-ade1-6b04120058d8-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.836312 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.836279    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e02684d-c911-4625-ade1-6b04120058d8-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.836586 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.836558    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzmnw\" (UniqueName: \"kubernetes.io/projected/3e02684d-c911-4625-ade1-6b04120058d8-kube-api-access-gzmnw\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-f226n\" (UID: \"3e02684d-c911-4625-ade1-6b04120058d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:34.886029 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.885145    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:52:34.934241 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:34.934189    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:35.431125 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:35.431094    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"]
Nov 02 19:52:35.432527 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:52:35.432491    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e02684d_c911_4625_ade1_6b04120058d8.slice/crio-c8c69fa3afaa94fbcdd19cc4f48589d1347dc58f2bfc78e0c907274fb8f2a37d WatchSource:0}: Error finding container c8c69fa3afaa94fbcdd19cc4f48589d1347dc58f2bfc78e0c907274fb8f2a37d: Status 404 returned error can't find the container with id c8c69fa3afaa94fbcdd19cc4f48589d1347dc58f2bfc78e0c907274fb8f2a37d
Nov 02 19:52:36.176333 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:36.176294    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n" event={"ID":"3e02684d-c911-4625-ade1-6b04120058d8","Type":"ContainerStarted","Data":"c8c69fa3afaa94fbcdd19cc4f48589d1347dc58f2bfc78e0c907274fb8f2a37d"}
Nov 02 19:52:38.517126 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:38.517025    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:52:40.198808 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:40.198757    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh" event={"ID":"c1097c45-59cb-4dc8-a416-989e95019e9b","Type":"ContainerStarted","Data":"cfc9958eea8d2dbddcab993dceffbdd3cd436de9c616fd0e18133b13be0e60b4"}
Nov 02 19:52:40.201552 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:40.201515    5224 generic.go:356] "Generic (PLEG): container finished" podID="29c18c81-02ef-422b-948c-7a9859862c2e" containerID="a1f661eaa3593d0a227c62a5bb3fcca7fbb85367615a6dc2befd0e1d8412657f" exitCode=0
Nov 02 19:52:40.201677 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:40.201597    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w" event={"ID":"29c18c81-02ef-422b-948c-7a9859862c2e","Type":"ContainerDied","Data":"a1f661eaa3593d0a227c62a5bb3fcca7fbb85367615a6dc2befd0e1d8412657f"}
Nov 02 19:52:40.203607 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:40.203556    5224 generic.go:356] "Generic (PLEG): container finished" podID="d2ca3136-91c1-499c-a96f-ea1328393a0b" containerID="c52b7f029e75d1a59a2ffc81eb7bf8987d16707d7b92790ebc0df8603cdfe5ca" exitCode=0
Nov 02 19:52:40.203683 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:40.203638    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p" event={"ID":"d2ca3136-91c1-499c-a96f-ea1328393a0b","Type":"ContainerDied","Data":"c52b7f029e75d1a59a2ffc81eb7bf8987d16707d7b92790ebc0df8603cdfe5ca"}
Nov 02 19:52:41.117184 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:41.117151    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 19:52:41.117280 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:41.117244    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 19:52:41.117316 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:41.117278    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 19:52:41.211855 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:41.211822    5224 generic.go:356] "Generic (PLEG): container finished" podID="9d018247-7db6-42ca-a57b-b53dc04c0e67" containerID="ccea1e4212c8dc710b220d1bb470621c3cb49944b1f8e1be27475247431585c4" exitCode=0
Nov 02 19:52:41.212322 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:41.211883    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n" event={"ID":"9d018247-7db6-42ca-a57b-b53dc04c0e67","Type":"ContainerDied","Data":"ccea1e4212c8dc710b220d1bb470621c3cb49944b1f8e1be27475247431585c4"}
Nov 02 19:52:41.213777 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:41.213737    5224 generic.go:356] "Generic (PLEG): container finished" podID="c1097c45-59cb-4dc8-a416-989e95019e9b" containerID="cfc9958eea8d2dbddcab993dceffbdd3cd436de9c616fd0e18133b13be0e60b4" exitCode=0
Nov 02 19:52:41.213904 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:41.213872    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh" event={"ID":"c1097c45-59cb-4dc8-a416-989e95019e9b","Type":"ContainerDied","Data":"cfc9958eea8d2dbddcab993dceffbdd3cd436de9c616fd0e18133b13be0e60b4"}
Nov 02 19:52:41.215858 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:41.215814    5224 generic.go:356] "Generic (PLEG): container finished" podID="62465b5f-e6e2-4734-8897-1c314c6429af" containerID="eff4ef85a297ad3c223136efa2b90defbb66f1ffa9d8ffdf497221d91965df85" exitCode=0
Nov 02 19:52:41.215928 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:41.215893    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk" event={"ID":"62465b5f-e6e2-4734-8897-1c314c6429af","Type":"ContainerDied","Data":"eff4ef85a297ad3c223136efa2b90defbb66f1ffa9d8ffdf497221d91965df85"}
Nov 02 19:52:41.217899 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:41.217879    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n" event={"ID":"3e02684d-c911-4625-ade1-6b04120058d8","Type":"ContainerStarted","Data":"51fe04d2264b85c088138d0171d1c529cb94a59f4005aef1b7fa7838e6dd0fa5"}
Nov 02 19:52:41.682719 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:41.682663    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n" podStartSLOduration=2.002938555 podStartE2EDuration="7.682648022s" totalImagesPullingTime="5.679709467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:41.68193845 +0000 UTC m=+242.679627059" watchObservedRunningTime="2026-11-02 19:52:41.682648022 +0000 UTC m=+242.680336630"
Nov 02 19:52:41.935162 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:41.935089    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:42.224816 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:42.224782    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w" event={"ID":"29c18c81-02ef-422b-948c-7a9859862c2e","Type":"ContainerStarted","Data":"dd69ef831cec0e37a7f7bb2179c5b4a6a570e5f50c55741577e674fd272d46ad"}
Nov 02 19:52:42.226844 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:42.226816    5224 generic.go:356] "Generic (PLEG): container finished" podID="d2ca3136-91c1-499c-a96f-ea1328393a0b" containerID="ce763cc965b9c13d0344d587968a61fc2f2b040fd34b032550dc2a04ce2ce2c4" exitCode=0
Nov 02 19:52:42.226959 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:42.226892    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p" event={"ID":"d2ca3136-91c1-499c-a96f-ea1328393a0b","Type":"ContainerDied","Data":"ce763cc965b9c13d0344d587968a61fc2f2b040fd34b032550dc2a04ce2ce2c4"}
Nov 02 19:52:42.939805 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:42.939760    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:43.234733 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:43.234605    5224 generic.go:356] "Generic (PLEG): container finished" podID="29c18c81-02ef-422b-948c-7a9859862c2e" containerID="dd69ef831cec0e37a7f7bb2179c5b4a6a570e5f50c55741577e674fd272d46ad" exitCode=0
Nov 02 19:52:43.234733 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:43.234717    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w" event={"ID":"29c18c81-02ef-422b-948c-7a9859862c2e","Type":"ContainerDied","Data":"dd69ef831cec0e37a7f7bb2179c5b4a6a570e5f50c55741577e674fd272d46ad"}
Nov 02 19:52:43.237249 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:43.237055    5224 generic.go:356] "Generic (PLEG): container finished" podID="c1097c45-59cb-4dc8-a416-989e95019e9b" containerID="5e44213b60798fe7dfaa41e8769a0a8055a7ea675bda9c17766ad992faea6d84" exitCode=0
Nov 02 19:52:43.237249 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:43.237185    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh" event={"ID":"c1097c45-59cb-4dc8-a416-989e95019e9b","Type":"ContainerDied","Data":"5e44213b60798fe7dfaa41e8769a0a8055a7ea675bda9c17766ad992faea6d84"}
Nov 02 19:52:43.238103 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:43.237801    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:43.238620 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:43.238552    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-f226n"
Nov 02 19:52:44.247061 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:44.246987    5224 generic.go:356] "Generic (PLEG): container finished" podID="62465b5f-e6e2-4734-8897-1c314c6429af" containerID="94fa0626c83fed3d9c7c8785ccde03002e93b61d78b078f5113080e313a6d814" exitCode=0
Nov 02 19:52:44.247589 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:44.247088    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk" event={"ID":"62465b5f-e6e2-4734-8897-1c314c6429af","Type":"ContainerDied","Data":"94fa0626c83fed3d9c7c8785ccde03002e93b61d78b078f5113080e313a6d814"}
Nov 02 19:52:46.040190 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:46.040048    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:52:46.263277 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:46.263025    5224 generic.go:356] "Generic (PLEG): container finished" podID="9d018247-7db6-42ca-a57b-b53dc04c0e67" containerID="857195744f41b5cc05d4439cf3c2466f686f99fb5c271f2b537b508473ea5255" exitCode=0
Nov 02 19:52:46.263277 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:46.263082    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n" event={"ID":"9d018247-7db6-42ca-a57b-b53dc04c0e67","Type":"ContainerDied","Data":"857195744f41b5cc05d4439cf3c2466f686f99fb5c271f2b537b508473ea5255"}
Nov 02 19:52:46.737670 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:46.737632    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:52:53.308418 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:53.308387    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p" event={"ID":"d2ca3136-91c1-499c-a96f-ea1328393a0b","Type":"ContainerStarted","Data":"f59056ebc4d22ec4395aac5d733e930a25fb4c94a0948b17f67fd23c41c62c3a"}
Nov 02 19:52:54.315114 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:54.315041    5224 generic.go:356] "Generic (PLEG): container finished" podID="9d018247-7db6-42ca-a57b-b53dc04c0e67" containerID="c4b7771ef1d0691d45f34098e49514278efcd581edcea988c6339dfc31e47668" exitCode=0
Nov 02 19:52:54.315576 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:54.315553    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n" event={"ID":"9d018247-7db6-42ca-a57b-b53dc04c0e67","Type":"ContainerDied","Data":"c4b7771ef1d0691d45f34098e49514278efcd581edcea988c6339dfc31e47668"}
Nov 02 19:52:54.317985 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:54.317958    5224 generic.go:356] "Generic (PLEG): container finished" podID="d2ca3136-91c1-499c-a96f-ea1328393a0b" containerID="f59056ebc4d22ec4395aac5d733e930a25fb4c94a0948b17f67fd23c41c62c3a" exitCode=0
Nov 02 19:52:54.318112 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:54.318005    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p" event={"ID":"d2ca3136-91c1-499c-a96f-ea1328393a0b","Type":"ContainerDied","Data":"f59056ebc4d22ec4395aac5d733e930a25fb4c94a0948b17f67fd23c41c62c3a"}
Nov 02 19:52:55.324859 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.324822    5224 generic.go:356] "Generic (PLEG): container finished" podID="29c18c81-02ef-422b-948c-7a9859862c2e" containerID="630bf2657a40191d0cf719684e165052e0555bb3a9d737b8e06ce8ee43b9c948" exitCode=0
Nov 02 19:52:55.325251 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.324884    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w" event={"ID":"29c18c81-02ef-422b-948c-7a9859862c2e","Type":"ContainerDied","Data":"630bf2657a40191d0cf719684e165052e0555bb3a9d737b8e06ce8ee43b9c948"}
Nov 02 19:52:55.327162 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.327136    5224 generic.go:356] "Generic (PLEG): container finished" podID="c1097c45-59cb-4dc8-a416-989e95019e9b" containerID="24b41a8def5e83e906b791fdbdcfdeb4d73ccdf614ad6c80a17b33a9846bcfbd" exitCode=0
Nov 02 19:52:55.327284 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.327219    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh" event={"ID":"c1097c45-59cb-4dc8-a416-989e95019e9b","Type":"ContainerDied","Data":"24b41a8def5e83e906b791fdbdcfdeb4d73ccdf614ad6c80a17b33a9846bcfbd"}
Nov 02 19:52:55.329225 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.329204    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk" event={"ID":"62465b5f-e6e2-4734-8897-1c314c6429af","Type":"ContainerStarted","Data":"8d53dab7a683b9c133489f8a5d2359e184388e6a81bc0c08d49e56435e8ad757"}
Nov 02 19:52:55.469421 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.469398    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"
Nov 02 19:52:55.500577 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.500543    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"
Nov 02 19:52:55.558003 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.557957    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk" podStartSLOduration=3.190255727 podStartE2EDuration="28.557943798s" totalImagesPullingTime="25.367688071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:55.557659025 +0000 UTC m=+256.555347635" watchObservedRunningTime="2026-11-02 19:52:55.557943798 +0000 UTC m=+256.555632407"
Nov 02 19:52:55.587677 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.587646    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-util\" (UniqueName: \"kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-util\") pod \"9d018247-7db6-42ca-a57b-b53dc04c0e67\" (UID: \"9d018247-7db6-42ca-a57b-b53dc04c0e67\") "
Nov 02 19:52:55.587869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.587691    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-bundle\") pod \"9d018247-7db6-42ca-a57b-b53dc04c0e67\" (UID: \"9d018247-7db6-42ca-a57b-b53dc04c0e67\") "
Nov 02 19:52:55.587869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.587761    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-util\" (UniqueName: \"kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-util\") pod \"d2ca3136-91c1-499c-a96f-ea1328393a0b\" (UID: \"d2ca3136-91c1-499c-a96f-ea1328393a0b\") "
Nov 02 19:52:55.587869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.587821    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ca3136-91c1-499c-a96f-ea1328393a0b-kube-api-access-b6xc4\" (UniqueName: \"kubernetes.io/projected/d2ca3136-91c1-499c-a96f-ea1328393a0b-kube-api-access-b6xc4\") pod \"d2ca3136-91c1-499c-a96f-ea1328393a0b\" (UID: \"d2ca3136-91c1-499c-a96f-ea1328393a0b\") "
Nov 02 19:52:55.587869 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.587855    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d018247-7db6-42ca-a57b-b53dc04c0e67-kube-api-access-xv4qh\" (UniqueName: \"kubernetes.io/projected/9d018247-7db6-42ca-a57b-b53dc04c0e67-kube-api-access-xv4qh\") pod \"9d018247-7db6-42ca-a57b-b53dc04c0e67\" (UID: \"9d018247-7db6-42ca-a57b-b53dc04c0e67\") "
Nov 02 19:52:55.588117 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.587895    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-bundle\") pod \"d2ca3136-91c1-499c-a96f-ea1328393a0b\" (UID: \"d2ca3136-91c1-499c-a96f-ea1328393a0b\") "
Nov 02 19:52:55.588303 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.588273    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-bundle" pod "9d018247-7db6-42ca-a57b-b53dc04c0e67" (UID: "9d018247-7db6-42ca-a57b-b53dc04c0e67"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:55.588522 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.588488    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-bundle" pod "d2ca3136-91c1-499c-a96f-ea1328393a0b" (UID: "d2ca3136-91c1-499c-a96f-ea1328393a0b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:55.588634 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.588557    5224 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-bundle\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:55.590410 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.590379    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d018247-7db6-42ca-a57b-b53dc04c0e67-kube-api-access-xv4qh" pod "9d018247-7db6-42ca-a57b-b53dc04c0e67" (UID: "9d018247-7db6-42ca-a57b-b53dc04c0e67"). InnerVolumeSpecName "kube-api-access-xv4qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:55.590529 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.590508    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ca3136-91c1-499c-a96f-ea1328393a0b-kube-api-access-b6xc4" pod "d2ca3136-91c1-499c-a96f-ea1328393a0b" (UID: "d2ca3136-91c1-499c-a96f-ea1328393a0b"). InnerVolumeSpecName "kube-api-access-b6xc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:55.592331 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.592309    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-util" pod "9d018247-7db6-42ca-a57b-b53dc04c0e67" (UID: "9d018247-7db6-42ca-a57b-b53dc04c0e67"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:55.594165 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.594146    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-util" pod "d2ca3136-91c1-499c-a96f-ea1328393a0b" (UID: "d2ca3136-91c1-499c-a96f-ea1328393a0b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:55.689816 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.689727    5224 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-util\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:55.689816 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.689757    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6xc4\" (UniqueName: \"kubernetes.io/projected/d2ca3136-91c1-499c-a96f-ea1328393a0b-kube-api-access-b6xc4\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:55.689994 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.689853    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv4qh\" (UniqueName: \"kubernetes.io/projected/9d018247-7db6-42ca-a57b-b53dc04c0e67-kube-api-access-xv4qh\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:55.689994 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.689871    5224 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2ca3136-91c1-499c-a96f-ea1328393a0b-bundle\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:55.689994 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:55.689884    5224 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d018247-7db6-42ca-a57b-b53dc04c0e67-util\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:56.335214 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.335182    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n"
Nov 02 19:52:56.335707 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.335180    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444mw4n" event={"ID":"9d018247-7db6-42ca-a57b-b53dc04c0e67","Type":"ContainerDied","Data":"fa67494ca8ddbdbf88aada084c4abad3fb9e8f6d47366a8b384f7802a7147dd6"}
Nov 02 19:52:56.335707 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.335344    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa67494ca8ddbdbf88aada084c4abad3fb9e8f6d47366a8b384f7802a7147dd6"
Nov 02 19:52:56.337520 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.337492    5224 generic.go:356] "Generic (PLEG): container finished" podID="62465b5f-e6e2-4734-8897-1c314c6429af" containerID="8d53dab7a683b9c133489f8a5d2359e184388e6a81bc0c08d49e56435e8ad757" exitCode=0
Nov 02 19:52:56.337654 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.337584    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk" event={"ID":"62465b5f-e6e2-4734-8897-1c314c6429af","Type":"ContainerDied","Data":"8d53dab7a683b9c133489f8a5d2359e184388e6a81bc0c08d49e56435e8ad757"}
Nov 02 19:52:56.339641 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.339612    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p" event={"ID":"d2ca3136-91c1-499c-a96f-ea1328393a0b","Type":"ContainerDied","Data":"75ed0088bd1d553e5494a163fe98a032574ebcc159e96dcf05b6e1850ff809df"}
Nov 02 19:52:56.339757 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.339651    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75ed0088bd1d553e5494a163fe98a032574ebcc159e96dcf05b6e1850ff809df"
Nov 02 19:52:56.339757 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.339663    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h9p"
Nov 02 19:52:56.500515 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.500487    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"
Nov 02 19:52:56.512547 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.512524    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"
Nov 02 19:52:56.597877 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.597791    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-util\" (UniqueName: \"kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-util\") pod \"c1097c45-59cb-4dc8-a416-989e95019e9b\" (UID: \"c1097c45-59cb-4dc8-a416-989e95019e9b\") "
Nov 02 19:52:56.597877 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.597866    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-bundle\") pod \"c1097c45-59cb-4dc8-a416-989e95019e9b\" (UID: \"c1097c45-59cb-4dc8-a416-989e95019e9b\") "
Nov 02 19:52:56.598107 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.597891    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-util\" (UniqueName: \"kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-util\") pod \"29c18c81-02ef-422b-948c-7a9859862c2e\" (UID: \"29c18c81-02ef-422b-948c-7a9859862c2e\") "
Nov 02 19:52:56.598107 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.597913    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-bundle\") pod \"29c18c81-02ef-422b-948c-7a9859862c2e\" (UID: \"29c18c81-02ef-422b-948c-7a9859862c2e\") "
Nov 02 19:52:56.598107 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.597940    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1097c45-59cb-4dc8-a416-989e95019e9b-kube-api-access-t4cm4\" (UniqueName: \"kubernetes.io/projected/c1097c45-59cb-4dc8-a416-989e95019e9b-kube-api-access-t4cm4\") pod \"c1097c45-59cb-4dc8-a416-989e95019e9b\" (UID: \"c1097c45-59cb-4dc8-a416-989e95019e9b\") "
Nov 02 19:52:56.598107 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.597997    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c18c81-02ef-422b-948c-7a9859862c2e-kube-api-access-7tp6c\" (UniqueName: \"kubernetes.io/projected/29c18c81-02ef-422b-948c-7a9859862c2e-kube-api-access-7tp6c\") pod \"29c18c81-02ef-422b-948c-7a9859862c2e\" (UID: \"29c18c81-02ef-422b-948c-7a9859862c2e\") "
Nov 02 19:52:56.598400 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.598374    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-bundle" pod "c1097c45-59cb-4dc8-a416-989e95019e9b" (UID: "c1097c45-59cb-4dc8-a416-989e95019e9b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:56.598589 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.598568    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-bundle" pod "29c18c81-02ef-422b-948c-7a9859862c2e" (UID: "29c18c81-02ef-422b-948c-7a9859862c2e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:56.600266 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.600236    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c18c81-02ef-422b-948c-7a9859862c2e-kube-api-access-7tp6c" pod "29c18c81-02ef-422b-948c-7a9859862c2e" (UID: "29c18c81-02ef-422b-948c-7a9859862c2e"). InnerVolumeSpecName "kube-api-access-7tp6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:56.600390 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.600366    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1097c45-59cb-4dc8-a416-989e95019e9b-kube-api-access-t4cm4" pod "c1097c45-59cb-4dc8-a416-989e95019e9b" (UID: "c1097c45-59cb-4dc8-a416-989e95019e9b"). InnerVolumeSpecName "kube-api-access-t4cm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:56.603987 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.603968    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-util" pod "c1097c45-59cb-4dc8-a416-989e95019e9b" (UID: "c1097c45-59cb-4dc8-a416-989e95019e9b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:56.605125 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.605103    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-util" pod "29c18c81-02ef-422b-948c-7a9859862c2e" (UID: "29c18c81-02ef-422b-948c-7a9859862c2e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:56.699335 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.699304    5224 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-util\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:56.699335 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.699330    5224 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1097c45-59cb-4dc8-a416-989e95019e9b-bundle\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:56.699335 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.699339    5224 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-util\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:56.699542 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.699347    5224 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29c18c81-02ef-422b-948c-7a9859862c2e-bundle\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:56.699542 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.699356    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4cm4\" (UniqueName: \"kubernetes.io/projected/c1097c45-59cb-4dc8-a416-989e95019e9b-kube-api-access-t4cm4\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:56.699542 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:56.699367    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tp6c\" (UniqueName: \"kubernetes.io/projected/29c18c81-02ef-422b-948c-7a9859862c2e-kube-api-access-7tp6c\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:57.346249 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.346214    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh" event={"ID":"c1097c45-59cb-4dc8-a416-989e95019e9b","Type":"ContainerDied","Data":"a87dcd794ead9f49e47c72320589daf7a248f391f40e667ed5ad8cbb470f5d2c"}
Nov 02 19:52:57.346676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.346275    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a87dcd794ead9f49e47c72320589daf7a248f391f40e667ed5ad8cbb470f5d2c"
Nov 02 19:52:57.346676 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.346231    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z46vh"
Nov 02 19:52:57.348617 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.348587    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w"
Nov 02 19:52:57.348753 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.348629    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847al2v8w" event={"ID":"29c18c81-02ef-422b-948c-7a9859862c2e","Type":"ContainerDied","Data":"4b789a3266692ff4c6ba4dbbdc4b8ec6f9ca1405ba0ae625259777d515cc1c13"}
Nov 02 19:52:57.348753 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.348662    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b789a3266692ff4c6ba4dbbdc4b8ec6f9ca1405ba0ae625259777d515cc1c13"
Nov 02 19:52:57.360706 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:52:57.360679    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:52:57.469188 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.469165    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"
Nov 02 19:52:57.607681 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.607653    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-util\" (UniqueName: \"kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-util\") pod \"62465b5f-e6e2-4734-8897-1c314c6429af\" (UID: \"62465b5f-e6e2-4734-8897-1c314c6429af\") "
Nov 02 19:52:57.607848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.607711    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62465b5f-e6e2-4734-8897-1c314c6429af-kube-api-access-6zxx4\" (UniqueName: \"kubernetes.io/projected/62465b5f-e6e2-4734-8897-1c314c6429af-kube-api-access-6zxx4\") pod \"62465b5f-e6e2-4734-8897-1c314c6429af\" (UID: \"62465b5f-e6e2-4734-8897-1c314c6429af\") "
Nov 02 19:52:57.607848 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.607759    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-bundle\" (UniqueName: \"kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-bundle\") pod \"62465b5f-e6e2-4734-8897-1c314c6429af\" (UID: \"62465b5f-e6e2-4734-8897-1c314c6429af\") "
Nov 02 19:52:57.608283 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.608252    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-bundle" pod "62465b5f-e6e2-4734-8897-1c314c6429af" (UID: "62465b5f-e6e2-4734-8897-1c314c6429af"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:57.609780 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.609747    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62465b5f-e6e2-4734-8897-1c314c6429af-kube-api-access-6zxx4" pod "62465b5f-e6e2-4734-8897-1c314c6429af" (UID: "62465b5f-e6e2-4734-8897-1c314c6429af"). InnerVolumeSpecName "kube-api-access-6zxx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:57.611585 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.611558    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-util" pod "62465b5f-e6e2-4734-8897-1c314c6429af" (UID: "62465b5f-e6e2-4734-8897-1c314c6429af"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:57.709380 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.709297    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxx4\" (UniqueName: \"kubernetes.io/projected/62465b5f-e6e2-4734-8897-1c314c6429af-kube-api-access-6zxx4\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:57.709380 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.709330    5224 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-bundle\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:57.709380 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:57.709342    5224 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62465b5f-e6e2-4734-8897-1c314c6429af-util\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:58.355057 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:58.355027    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk"
Nov 02 19:52:58.355057 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:58.355044    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7wgk" event={"ID":"62465b5f-e6e2-4734-8897-1c314c6429af","Type":"ContainerDied","Data":"da95a5a732b91158f4752dacdc0ea030ed608e6ddcf47d6951766592a1029d59"}
Nov 02 19:52:58.355482 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:52:58.355099    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da95a5a732b91158f4752dacdc0ea030ed608e6ddcf47d6951766592a1029d59"
Nov 02 19:53:02.138245 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:02.138211    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:02.442188 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:02.442091    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:04.064640 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064606    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj"]
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064907    5224 cpu_manager.go:414] "containerMap: removing container" podUID="29c18c81-02ef-422b-948c-7a9859862c2e" containerName="pull"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064918    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c18c81-02ef-422b-948c-7a9859862c2e" containerName="pull"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064928    5224 cpu_manager.go:414] "containerMap: removing container" podUID="62465b5f-e6e2-4734-8897-1c314c6429af" containerName="pull"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064934    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62465b5f-e6e2-4734-8897-1c314c6429af" containerName="pull"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064945    5224 cpu_manager.go:414] "containerMap: removing container" podUID="29c18c81-02ef-422b-948c-7a9859862c2e" containerName="extract"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064950    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c18c81-02ef-422b-948c-7a9859862c2e" containerName="extract"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064960    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d2ca3136-91c1-499c-a96f-ea1328393a0b" containerName="pull"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064965    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ca3136-91c1-499c-a96f-ea1328393a0b" containerName="pull"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064973    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d2ca3136-91c1-499c-a96f-ea1328393a0b" containerName="util"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064978    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ca3136-91c1-499c-a96f-ea1328393a0b" containerName="util"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064983    5224 cpu_manager.go:414] "containerMap: removing container" podUID="c1097c45-59cb-4dc8-a416-989e95019e9b" containerName="util"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064987    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1097c45-59cb-4dc8-a416-989e95019e9b" containerName="util"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064995    5224 cpu_manager.go:414] "containerMap: removing container" podUID="29c18c81-02ef-422b-948c-7a9859862c2e" containerName="util"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.064999    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c18c81-02ef-422b-948c-7a9859862c2e" containerName="util"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065005    5224 cpu_manager.go:414] "containerMap: removing container" podUID="c1097c45-59cb-4dc8-a416-989e95019e9b" containerName="pull"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065010    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1097c45-59cb-4dc8-a416-989e95019e9b" containerName="pull"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065015    5224 cpu_manager.go:414] "containerMap: removing container" podUID="9d018247-7db6-42ca-a57b-b53dc04c0e67" containerName="pull"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065019    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d018247-7db6-42ca-a57b-b53dc04c0e67" containerName="pull"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065032    5224 cpu_manager.go:414] "containerMap: removing container" podUID="9d018247-7db6-42ca-a57b-b53dc04c0e67" containerName="util"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065036    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d018247-7db6-42ca-a57b-b53dc04c0e67" containerName="util"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065042    5224 cpu_manager.go:414] "containerMap: removing container" podUID="62465b5f-e6e2-4734-8897-1c314c6429af" containerName="extract"
Nov 02 19:53:04.065034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065047    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62465b5f-e6e2-4734-8897-1c314c6429af" containerName="extract"
Nov 02 19:53:04.065644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065056    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d2ca3136-91c1-499c-a96f-ea1328393a0b" containerName="extract"
Nov 02 19:53:04.065644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065060    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ca3136-91c1-499c-a96f-ea1328393a0b" containerName="extract"
Nov 02 19:53:04.065644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065067    5224 cpu_manager.go:414] "containerMap: removing container" podUID="62465b5f-e6e2-4734-8897-1c314c6429af" containerName="util"
Nov 02 19:53:04.065644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065071    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62465b5f-e6e2-4734-8897-1c314c6429af" containerName="util"
Nov 02 19:53:04.065644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065077    5224 cpu_manager.go:414] "containerMap: removing container" podUID="c1097c45-59cb-4dc8-a416-989e95019e9b" containerName="extract"
Nov 02 19:53:04.065644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065081    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1097c45-59cb-4dc8-a416-989e95019e9b" containerName="extract"
Nov 02 19:53:04.065644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065091    5224 cpu_manager.go:414] "containerMap: removing container" podUID="9d018247-7db6-42ca-a57b-b53dc04c0e67" containerName="extract"
Nov 02 19:53:04.065644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065096    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d018247-7db6-42ca-a57b-b53dc04c0e67" containerName="extract"
Nov 02 19:53:04.065644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065135    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d018247-7db6-42ca-a57b-b53dc04c0e67" containerName="extract"
Nov 02 19:53:04.065644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065142    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c18c81-02ef-422b-948c-7a9859862c2e" containerName="extract"
Nov 02 19:53:04.065644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065151    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="62465b5f-e6e2-4734-8897-1c314c6429af" containerName="extract"
Nov 02 19:53:04.065644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065157    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ca3136-91c1-499c-a96f-ea1328393a0b" containerName="extract"
Nov 02 19:53:04.065644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.065162    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1097c45-59cb-4dc8-a416-989e95019e9b" containerName="extract"
Nov 02 19:53:04.067136 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.067120    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj"
Nov 02 19:53:04.121829 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.121797    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj"]
Nov 02 19:53:04.162751 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.162719    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/344e51a0-0b6a-4c3c-aeb9-12d962d62ded-tmp\") pod \"jobset-operator-674cd59dc5-wf5tj\" (UID: \"344e51a0-0b6a-4c3c-aeb9-12d962d62ded\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj"
Nov 02 19:53:04.162941 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.162874    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sg9b\" (UniqueName: \"kubernetes.io/projected/344e51a0-0b6a-4c3c-aeb9-12d962d62ded-kube-api-access-4sg9b\") pod \"jobset-operator-674cd59dc5-wf5tj\" (UID: \"344e51a0-0b6a-4c3c-aeb9-12d962d62ded\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj"
Nov 02 19:53:04.172038 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.172013    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:53:04.172038 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.172028    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-zqlcb\""
Nov 02 19:53:04.172189 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.172029    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 19:53:04.263588 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.263552    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/344e51a0-0b6a-4c3c-aeb9-12d962d62ded-tmp\") pod \"jobset-operator-674cd59dc5-wf5tj\" (UID: \"344e51a0-0b6a-4c3c-aeb9-12d962d62ded\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj"
Nov 02 19:53:04.263739 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.263615    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sg9b\" (UniqueName: \"kubernetes.io/projected/344e51a0-0b6a-4c3c-aeb9-12d962d62ded-kube-api-access-4sg9b\") pod \"jobset-operator-674cd59dc5-wf5tj\" (UID: \"344e51a0-0b6a-4c3c-aeb9-12d962d62ded\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj"
Nov 02 19:53:04.263983 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.263962    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/344e51a0-0b6a-4c3c-aeb9-12d962d62ded-tmp\") pod \"jobset-operator-674cd59dc5-wf5tj\" (UID: \"344e51a0-0b6a-4c3c-aeb9-12d962d62ded\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj"
Nov 02 19:53:04.322252 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.322181    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sg9b\" (UniqueName: \"kubernetes.io/projected/344e51a0-0b6a-4c3c-aeb9-12d962d62ded-kube-api-access-4sg9b\") pod \"jobset-operator-674cd59dc5-wf5tj\" (UID: \"344e51a0-0b6a-4c3c-aeb9-12d962d62ded\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj"
Nov 02 19:53:04.390528 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.390491    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj"
Nov 02 19:53:04.752392 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:53:04.752356    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod344e51a0_0b6a_4c3c_aeb9_12d962d62ded.slice/crio-4ede2dbe1d16376381f161e5e82d6b1a1b6703d33350eba82a992d1f4bc55b78 WatchSource:0}: Error finding container 4ede2dbe1d16376381f161e5e82d6b1a1b6703d33350eba82a992d1f4bc55b78: Status 404 returned error can't find the container with id 4ede2dbe1d16376381f161e5e82d6b1a1b6703d33350eba82a992d1f4bc55b78
Nov 02 19:53:04.754489 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:04.754466    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj"]
Nov 02 19:53:05.383386 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:05.383353    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj" event={"ID":"344e51a0-0b6a-4c3c-aeb9-12d962d62ded","Type":"ContainerStarted","Data":"4ede2dbe1d16376381f161e5e82d6b1a1b6703d33350eba82a992d1f4bc55b78"}
Nov 02 19:53:11.411037 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:11.411001    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj" event={"ID":"344e51a0-0b6a-4c3c-aeb9-12d962d62ded","Type":"ContainerStarted","Data":"49effaa5d7cf9caf6c05708fbf9178fa5708bcae8cb92c4ca666910880158a63"}
Nov 02 19:53:11.527822 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:11.527752    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wf5tj" podStartSLOduration=1.453849825 podStartE2EDuration="7.527734776s" totalImagesPullingTime="6.073884951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:11.526923354 +0000 UTC m=+272.524611975" watchObservedRunningTime="2026-11-02 19:53:11.527734776 +0000 UTC m=+272.525423387"
Nov 02 19:53:12.552374 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:12.552330    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:12.809386 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:12.809313    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:13.783987 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:13.783946    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:13.932080 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:13.932053    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"]
Nov 02 19:53:13.934715 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:13.934693    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.034082 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.034005    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c49ae9da-2414-4d5a-9a65-71dffbebf54f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.034082 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.034036    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c49ae9da-2414-4d5a-9a65-71dffbebf54f-manifests\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.034082 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.034059    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxfb\" (UniqueName: \"kubernetes.io/projected/c49ae9da-2414-4d5a-9a65-71dffbebf54f-kube-api-access-twxfb\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.034279 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.034212    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c49ae9da-2414-4d5a-9a65-71dffbebf54f-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.034279 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.034245    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c49ae9da-2414-4d5a-9a65-71dffbebf54f-tmp\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.034279 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.034266    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c49ae9da-2414-4d5a-9a65-71dffbebf54f-metrics-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.044031 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.044008    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"]
Nov 02 19:53:14.050477 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.050450    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 19:53:14.050563 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.050450    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 19:53:14.050563 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.050520    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 19:53:14.050563 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.050534    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 19:53:14.050674 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.050605    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4tm9h\""
Nov 02 19:53:14.051731 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.051713    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 19:53:14.118226 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.118192    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:14.135145 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.135123    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c49ae9da-2414-4d5a-9a65-71dffbebf54f-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.135287 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.135156    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c49ae9da-2414-4d5a-9a65-71dffbebf54f-tmp\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.135287 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.135181    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c49ae9da-2414-4d5a-9a65-71dffbebf54f-metrics-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.135287 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.135221    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c49ae9da-2414-4d5a-9a65-71dffbebf54f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.135440 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.135347    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c49ae9da-2414-4d5a-9a65-71dffbebf54f-manifests\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.135440 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.135398    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxfb\" (UniqueName: \"kubernetes.io/projected/c49ae9da-2414-4d5a-9a65-71dffbebf54f-kube-api-access-twxfb\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.135701 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.135676    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c49ae9da-2414-4d5a-9a65-71dffbebf54f-manifests\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.137603 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.137571    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c49ae9da-2414-4d5a-9a65-71dffbebf54f-tmp\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.137790 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.137749    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c49ae9da-2414-4d5a-9a65-71dffbebf54f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.137855 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.137805    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c49ae9da-2414-4d5a-9a65-71dffbebf54f-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.137949 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.137930    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c49ae9da-2414-4d5a-9a65-71dffbebf54f-metrics-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.201718 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.201680    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxfb\" (UniqueName: \"kubernetes.io/projected/c49ae9da-2414-4d5a-9a65-71dffbebf54f-kube-api-access-twxfb\") pod \"opendatahub-operator-controller-manager-55d64b46df-hr6tt\" (UID: \"c49ae9da-2414-4d5a-9a65-71dffbebf54f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.247758 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.247728    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:14.355037 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.355008    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:14.572754 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.572661    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:14.715755 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:53:14.715715    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49ae9da_2414_4d5a_9a65_71dffbebf54f.slice/crio-754bc31cb88f7a685391e86d3895fcf447ab25ae4baefc381e77013b0ba38472 WatchSource:0}: Error finding container 754bc31cb88f7a685391e86d3895fcf447ab25ae4baefc381e77013b0ba38472: Status 404 returned error can't find the container with id 754bc31cb88f7a685391e86d3895fcf447ab25ae4baefc381e77013b0ba38472
Nov 02 19:53:14.718778 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:14.718746    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"]
Nov 02 19:53:15.428977 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:15.428940    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt" event={"ID":"c49ae9da-2414-4d5a-9a65-71dffbebf54f","Type":"ContainerStarted","Data":"754bc31cb88f7a685391e86d3895fcf447ab25ae4baefc381e77013b0ba38472"}
Nov 02 19:53:17.537504 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:17.537469    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:17.843575 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:17.843485    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:18.445016 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:18.444979    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt" event={"ID":"c49ae9da-2414-4d5a-9a65-71dffbebf54f","Type":"ContainerStarted","Data":"70650d43b4329dcb0368a6fd395923786306155cab4f088d324752b92ed331a5"}
Nov 02 19:53:19.450438 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:19.450406    5224 generic.go:356] "Generic (PLEG): container finished" podID="c49ae9da-2414-4d5a-9a65-71dffbebf54f" containerID="70650d43b4329dcb0368a6fd395923786306155cab4f088d324752b92ed331a5" exitCode=0
Nov 02 19:53:19.450844 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:19.450505    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt" event={"ID":"c49ae9da-2414-4d5a-9a65-71dffbebf54f","Type":"ContainerDied","Data":"70650d43b4329dcb0368a6fd395923786306155cab4f088d324752b92ed331a5"}
Nov 02 19:53:19.778985 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:19.778951    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:20.456485 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:20.456454    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt" event={"ID":"c49ae9da-2414-4d5a-9a65-71dffbebf54f","Type":"ContainerStarted","Data":"6dccbdf6428a306139fe89a36a2406b8cbdf78f3609e1b6487807046147fc090"}
Nov 02 19:53:20.456887 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:20.456555    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:20.697926 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:20.697830    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt" podStartSLOduration=2.8720885689999998 podStartE2EDuration="7.697815873s" totalImagesPullingTime="4.350470812s" totalInitContainerRuntime="475.256492ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:20.646158759 +0000 UTC m=+281.643847379" watchObservedRunningTime="2026-11-02 19:53:20.697815873 +0000 UTC m=+281.695504479"
Nov 02 19:53:21.722740 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:21.722701    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:22.743229 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:22.743193    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:22.953354 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:22.953320    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:23.141744 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:23.141716    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:23.736521 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:23.736487    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:24.135063 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.135027    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn"]
Nov 02 19:53:24.158117 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.158095    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn"
Nov 02 19:53:24.191583 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.191556    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn"]
Nov 02 19:53:24.212303 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.212281    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-xs9cx\""
Nov 02 19:53:24.213552 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.213529    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9e09540-62b0-4bc8-b618-424f8f58c2a7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kkddn\" (UID: \"f9e09540-62b0-4bc8-b618-424f8f58c2a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn"
Nov 02 19:53:24.213700 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.213605    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwlk\" (UniqueName: \"kubernetes.io/projected/f9e09540-62b0-4bc8-b618-424f8f58c2a7-kube-api-access-wcwlk\") pod \"cert-manager-webhook-5dbbbf75cc-kkddn\" (UID: \"f9e09540-62b0-4bc8-b618-424f8f58c2a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn"
Nov 02 19:53:24.241851 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.241823    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 19:53:24.241953 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.241851    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 19:53:24.321317 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.317311    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwlk\" (UniqueName: \"kubernetes.io/projected/f9e09540-62b0-4bc8-b618-424f8f58c2a7-kube-api-access-wcwlk\") pod \"cert-manager-webhook-5dbbbf75cc-kkddn\" (UID: \"f9e09540-62b0-4bc8-b618-424f8f58c2a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn"
Nov 02 19:53:24.321480 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.321435    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9e09540-62b0-4bc8-b618-424f8f58c2a7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kkddn\" (UID: \"f9e09540-62b0-4bc8-b618-424f8f58c2a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn"
Nov 02 19:53:24.376375 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.376350    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwlk\" (UniqueName: \"kubernetes.io/projected/f9e09540-62b0-4bc8-b618-424f8f58c2a7-kube-api-access-wcwlk\") pod \"cert-manager-webhook-5dbbbf75cc-kkddn\" (UID: \"f9e09540-62b0-4bc8-b618-424f8f58c2a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn"
Nov 02 19:53:24.381732 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.381703    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9e09540-62b0-4bc8-b618-424f8f58c2a7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kkddn\" (UID: \"f9e09540-62b0-4bc8-b618-424f8f58c2a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn"
Nov 02 19:53:24.387892 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.387814    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:24.479343 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.479317    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn"
Nov 02 19:53:24.857307 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:53:24.857268    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9e09540_62b0_4bc8_b618_424f8f58c2a7.slice/crio-ddb8240ef1ba3ccbbc172c8b7e14f4329e429f52ddc9b448311a23bc680cce7f WatchSource:0}: Error finding container ddb8240ef1ba3ccbbc172c8b7e14f4329e429f52ddc9b448311a23bc680cce7f: Status 404 returned error can't find the container with id ddb8240ef1ba3ccbbc172c8b7e14f4329e429f52ddc9b448311a23bc680cce7f
Nov 02 19:53:24.859863 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:24.859841    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn"]
Nov 02 19:53:25.477486 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:25.477452    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn" event={"ID":"f9e09540-62b0-4bc8-b618-424f8f58c2a7","Type":"ContainerStarted","Data":"ddb8240ef1ba3ccbbc172c8b7e14f4329e429f52ddc9b448311a23bc680cce7f"}
Nov 02 19:53:25.950263 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:25.950229    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:27.358357 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:53:27.358324    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:53:28.304711 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:28.304674    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:28.546034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:28.545995    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:30.501918 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:30.501870    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn" event={"ID":"f9e09540-62b0-4bc8-b618-424f8f58c2a7","Type":"ContainerStarted","Data":"97c2a4851f7209c486c3e87c2da12d58e497042c444e3baa97679859e87c3137"}
Nov 02 19:53:30.502312 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:30.501987    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn"
Nov 02 19:53:30.544303 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:30.544267    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:30.622363 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:30.622307    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn" podStartSLOduration=1.630698158 podStartE2EDuration="6.622292613s" totalImagesPullingTime="4.991594455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:30.621815675 +0000 UTC m=+291.619504289" watchObservedRunningTime="2026-11-02 19:53:30.622292613 +0000 UTC m=+291.619981222"
Nov 02 19:53:30.789078 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:30.789027    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:30.976776 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:30.976739    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:31.462786 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:31.462743    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-hr6tt"
Nov 02 19:53:32.357835 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:32.357802    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:32.654070 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:32.653989    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k"]
Nov 02 19:53:32.656438 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:32.656422    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k"
Nov 02 19:53:32.695074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:32.695049    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjs48\" (UniqueName: \"kubernetes.io/projected/69235044-5904-4cad-93db-315e6e36f8fd-kube-api-access-qjs48\") pod \"limitador-operator-controller-manager-5778bb4449-rkw4k\" (UID: \"69235044-5904-4cad-93db-315e6e36f8fd\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k"
Nov 02 19:53:32.719882 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:32.719848    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 19:53:32.720199 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:32.720182    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k"]
Nov 02 19:53:32.722983 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:32.722966    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wkw2l\""
Nov 02 19:53:32.795547 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:32.795514    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjs48\" (UniqueName: \"kubernetes.io/projected/69235044-5904-4cad-93db-315e6e36f8fd-kube-api-access-qjs48\") pod \"limitador-operator-controller-manager-5778bb4449-rkw4k\" (UID: \"69235044-5904-4cad-93db-315e6e36f8fd\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k"
Nov 02 19:53:32.817084 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:32.817059    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 19:53:32.853946 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:32.853912    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjs48\" (UniqueName: \"kubernetes.io/projected/69235044-5904-4cad-93db-315e6e36f8fd-kube-api-access-qjs48\") pod \"limitador-operator-controller-manager-5778bb4449-rkw4k\" (UID: \"69235044-5904-4cad-93db-315e6e36f8fd\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k"
Nov 02 19:53:32.968084 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:32.967999    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k"
Nov 02 19:53:33.288888 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:33.288850    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:33.329493 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:53:33.329462    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69235044_5904_4cad_93db_315e6e36f8fd.slice/crio-9d62a4c9c143953b9f3a7cdbb3211d361d6284ebac97df802782541a936bda6d WatchSource:0}: Error finding container 9d62a4c9c143953b9f3a7cdbb3211d361d6284ebac97df802782541a936bda6d: Status 404 returned error can't find the container with id 9d62a4c9c143953b9f3a7cdbb3211d361d6284ebac97df802782541a936bda6d
Nov 02 19:53:33.331353 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:33.331330    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k"]
Nov 02 19:53:33.515255 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:33.515220    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k" event={"ID":"69235044-5904-4cad-93db-315e6e36f8fd","Type":"ContainerStarted","Data":"9d62a4c9c143953b9f3a7cdbb3211d361d6284ebac97df802782541a936bda6d"}
Nov 02 19:53:34.163177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:34.163145    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:34.398347 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:34.398318    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:36.275186 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:36.275151    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-frh4g"]
Nov 02 19:53:36.277424 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:36.277400    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-frh4g"
Nov 02 19:53:36.327172 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:36.327025    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3405ec4e-9918-40dc-916a-55198efce6e6-bound-sa-token\") pod \"cert-manager-597b95cb84-frh4g\" (UID: \"3405ec4e-9918-40dc-916a-55198efce6e6\") " pod="cert-manager/cert-manager-597b95cb84-frh4g"
Nov 02 19:53:36.327579 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:36.327552    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xqn\" (UniqueName: \"kubernetes.io/projected/3405ec4e-9918-40dc-916a-55198efce6e6-kube-api-access-n5xqn\") pod \"cert-manager-597b95cb84-frh4g\" (UID: \"3405ec4e-9918-40dc-916a-55198efce6e6\") " pod="cert-manager/cert-manager-597b95cb84-frh4g"
Nov 02 19:53:36.382318 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:36.382286    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2pwdv\""
Nov 02 19:53:36.383751 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:36.383720    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-frh4g"]
Nov 02 19:53:36.428461 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:36.428427    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xqn\" (UniqueName: \"kubernetes.io/projected/3405ec4e-9918-40dc-916a-55198efce6e6-kube-api-access-n5xqn\") pod \"cert-manager-597b95cb84-frh4g\" (UID: \"3405ec4e-9918-40dc-916a-55198efce6e6\") " pod="cert-manager/cert-manager-597b95cb84-frh4g"
Nov 02 19:53:36.428644 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:36.428490    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3405ec4e-9918-40dc-916a-55198efce6e6-bound-sa-token\") pod \"cert-manager-597b95cb84-frh4g\" (UID: \"3405ec4e-9918-40dc-916a-55198efce6e6\") " pod="cert-manager/cert-manager-597b95cb84-frh4g"
Nov 02 19:53:36.486882 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:36.486848    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3405ec4e-9918-40dc-916a-55198efce6e6-bound-sa-token\") pod \"cert-manager-597b95cb84-frh4g\" (UID: \"3405ec4e-9918-40dc-916a-55198efce6e6\") " pod="cert-manager/cert-manager-597b95cb84-frh4g"
Nov 02 19:53:36.487107 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:36.487071    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xqn\" (UniqueName: \"kubernetes.io/projected/3405ec4e-9918-40dc-916a-55198efce6e6-kube-api-access-n5xqn\") pod \"cert-manager-597b95cb84-frh4g\" (UID: \"3405ec4e-9918-40dc-916a-55198efce6e6\") " pod="cert-manager/cert-manager-597b95cb84-frh4g"
Nov 02 19:53:36.508845 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:36.508820    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kkddn"
Nov 02 19:53:36.589149 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:36.589069    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-frh4g"
Nov 02 19:53:36.975899 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:53:36.975871    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3405ec4e_9918_40dc_916a_55198efce6e6.slice/crio-3f5b23143c3cd978e11bed78c2029034b8503ad5dc4b73309944d19a1e117f44 WatchSource:0}: Error finding container 3f5b23143c3cd978e11bed78c2029034b8503ad5dc4b73309944d19a1e117f44: Status 404 returned error can't find the container with id 3f5b23143c3cd978e11bed78c2029034b8503ad5dc4b73309944d19a1e117f44
Nov 02 19:53:36.978212 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:36.978192    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-frh4g"]
Nov 02 19:53:37.535417 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:37.535376    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-frh4g" event={"ID":"3405ec4e-9918-40dc-916a-55198efce6e6","Type":"ContainerStarted","Data":"a966285d588da04a3c007052e9fc09cea07b0c3da5f2084f8ad70afcb8e0736b"}
Nov 02 19:53:37.535417 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:37.535431    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-frh4g" event={"ID":"3405ec4e-9918-40dc-916a-55198efce6e6","Type":"ContainerStarted","Data":"3f5b23143c3cd978e11bed78c2029034b8503ad5dc4b73309944d19a1e117f44"}
Nov 02 19:53:37.537056 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:37.537024    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k" event={"ID":"69235044-5904-4cad-93db-315e6e36f8fd","Type":"ContainerStarted","Data":"cf5f0add92e0490c31c5786b0b4c52f8de3464c60bdab8279291f7f013468442"}
Nov 02 19:53:37.537177 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:37.537160    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k"
Nov 02 19:53:37.661613 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:37.661516    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-frh4g" podStartSLOduration=1.661503412 podStartE2EDuration="1.661503412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:37.660542824 +0000 UTC m=+298.658231432" watchObservedRunningTime="2026-11-02 19:53:37.661503412 +0000 UTC m=+298.659192021"
Nov 02 19:53:37.783317 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:37.783264    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k" podStartSLOduration=1.8052370020000001 podStartE2EDuration="5.783250174s" totalImagesPullingTime="3.978013172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:37.782166245 +0000 UTC m=+298.779854854" watchObservedRunningTime="2026-11-02 19:53:37.783250174 +0000 UTC m=+298.780938783"
Nov 02 19:53:38.104658 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:38.104624    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:38.532083 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:38.532045    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:40.648362 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:40.648321    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:40.882472 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:40.882433    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:41.081401 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:41.081358    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:44.503664 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:44.503626    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:44.731257 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:44.731225    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:44.909865 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:44.909831    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:45.087009 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:45.086973    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:45.303934 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:45.303898    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:45.546752 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:45.546714    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:46.354363 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:46.354332    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:47.443915 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:47.443880    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:47.859180 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:47.859141    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:48.122119 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:48.122043    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:48.380515 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:48.380408    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:48.543358 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:48.543327    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k"
Nov 02 19:53:49.817208 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:49.817173    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:49.825110 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:49.825086    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"]
Nov 02 19:53:49.828456 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:49.828441    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"
Nov 02 19:53:49.937619 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:49.937580    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 19:53:49.937619 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:49.937614    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mqm8f\""
Nov 02 19:53:49.937863 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:49.937697    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 19:53:49.938866 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:49.938846    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"]
Nov 02 19:53:49.943587 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:49.943561    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e59fe7-4223-4124-ba42-c5dde81b4c72-tls-certs\") pod \"kube-auth-proxy-7c9c5d7d8f-g4db8\" (UID: \"a3e59fe7-4223-4124-ba42-c5dde81b4c72\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"
Nov 02 19:53:49.943721 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:49.943641    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7csw7\" (UniqueName: \"kubernetes.io/projected/a3e59fe7-4223-4124-ba42-c5dde81b4c72-kube-api-access-7csw7\") pod \"kube-auth-proxy-7c9c5d7d8f-g4db8\" (UID: \"a3e59fe7-4223-4124-ba42-c5dde81b4c72\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"
Nov 02 19:53:49.943721 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:49.943694    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3e59fe7-4223-4124-ba42-c5dde81b4c72-tmp\") pod \"kube-auth-proxy-7c9c5d7d8f-g4db8\" (UID: \"a3e59fe7-4223-4124-ba42-c5dde81b4c72\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"
Nov 02 19:53:50.010068 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.010034    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:50.045123 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.045089    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7csw7\" (UniqueName: \"kubernetes.io/projected/a3e59fe7-4223-4124-ba42-c5dde81b4c72-kube-api-access-7csw7\") pod \"kube-auth-proxy-7c9c5d7d8f-g4db8\" (UID: \"a3e59fe7-4223-4124-ba42-c5dde81b4c72\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"
Nov 02 19:53:50.045232 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.045146    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3e59fe7-4223-4124-ba42-c5dde81b4c72-tmp\") pod \"kube-auth-proxy-7c9c5d7d8f-g4db8\" (UID: \"a3e59fe7-4223-4124-ba42-c5dde81b4c72\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"
Nov 02 19:53:50.045232 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.045191    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e59fe7-4223-4124-ba42-c5dde81b4c72-tls-certs\") pod \"kube-auth-proxy-7c9c5d7d8f-g4db8\" (UID: \"a3e59fe7-4223-4124-ba42-c5dde81b4c72\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"
Nov 02 19:53:50.047461 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.047432    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3e59fe7-4223-4124-ba42-c5dde81b4c72-tmp\") pod \"kube-auth-proxy-7c9c5d7d8f-g4db8\" (UID: \"a3e59fe7-4223-4124-ba42-c5dde81b4c72\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"
Nov 02 19:53:50.047703 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.047682    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e59fe7-4223-4124-ba42-c5dde81b4c72-tls-certs\") pod \"kube-auth-proxy-7c9c5d7d8f-g4db8\" (UID: \"a3e59fe7-4223-4124-ba42-c5dde81b4c72\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"
Nov 02 19:53:50.106317 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.106246    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7csw7\" (UniqueName: \"kubernetes.io/projected/a3e59fe7-4223-4124-ba42-c5dde81b4c72-kube-api-access-7csw7\") pod \"kube-auth-proxy-7c9c5d7d8f-g4db8\" (UID: \"a3e59fe7-4223-4124-ba42-c5dde81b4c72\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"
Nov 02 19:53:50.139113 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.139073    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"
Nov 02 19:53:50.362293 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.362182    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:50.506020 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:53:50.505987    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e59fe7_4223_4124_ba42_c5dde81b4c72.slice/crio-46668402078e7046482ca8df18c65df3698bba1826a6e1c2c87317c96ee42402 WatchSource:0}: Error finding container 46668402078e7046482ca8df18c65df3698bba1826a6e1c2c87317c96ee42402: Status 404 returned error can't find the container with id 46668402078e7046482ca8df18c65df3698bba1826a6e1c2c87317c96ee42402
Nov 02 19:53:50.508161 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.508138    5224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:53:50.508722 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.508703    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8"]
Nov 02 19:53:50.593524 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.593487    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8" event={"ID":"a3e59fe7-4223-4124-ba42-c5dde81b4c72","Type":"ContainerStarted","Data":"46668402078e7046482ca8df18c65df3698bba1826a6e1c2c87317c96ee42402"}
Nov 02 19:53:50.647724 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.647619    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:50.950332 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:50.950222    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:51.724326 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:51.724289    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:51.901358 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:51.901322    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:52.184585 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:52.184544    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:52.372430 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:52.372393    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:52.556882 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:52.556846    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:52.770314 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:52.770280    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:53.634465 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:53.634441    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2hhpv"]
Nov 02 19:53:53.637694 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:53.637672    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2hhpv"
Nov 02 19:53:53.689453 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:53.689419    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2hhpv"]
Nov 02 19:53:53.692881 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:53.692858    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vvsx7\""
Nov 02 19:53:53.780874 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:53.780831    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89kp5\" (UniqueName: \"kubernetes.io/projected/27c5829b-2a4f-4ddd-b413-c8d8268ff8c1-kube-api-access-89kp5\") pod \"authorino-57f7f8659f-2hhpv\" (UID: \"27c5829b-2a4f-4ddd-b413-c8d8268ff8c1\") " pod="kuadrant-system/authorino-57f7f8659f-2hhpv"
Nov 02 19:53:53.815236 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:53.815201    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:53.881349 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:53.881280    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89kp5\" (UniqueName: \"kubernetes.io/projected/27c5829b-2a4f-4ddd-b413-c8d8268ff8c1-kube-api-access-89kp5\") pod \"authorino-57f7f8659f-2hhpv\" (UID: \"27c5829b-2a4f-4ddd-b413-c8d8268ff8c1\") " pod="kuadrant-system/authorino-57f7f8659f-2hhpv"
Nov 02 19:53:53.940531 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:53.940490    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89kp5\" (UniqueName: \"kubernetes.io/projected/27c5829b-2a4f-4ddd-b413-c8d8268ff8c1-kube-api-access-89kp5\") pod \"authorino-57f7f8659f-2hhpv\" (UID: \"27c5829b-2a4f-4ddd-b413-c8d8268ff8c1\") " pod="kuadrant-system/authorino-57f7f8659f-2hhpv"
Nov 02 19:53:53.949021 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:53.949000    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2hhpv"
Nov 02 19:53:54.039395 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:54.039343    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:54.308653 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:53:54.308618    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27c5829b_2a4f_4ddd_b413_c8d8268ff8c1.slice/crio-276b85a8e84611cd7986f93c59222b0b90a237f2613935c4c7ccd79ef124fa73 WatchSource:0}: Error finding container 276b85a8e84611cd7986f93c59222b0b90a237f2613935c4c7ccd79ef124fa73: Status 404 returned error can't find the container with id 276b85a8e84611cd7986f93c59222b0b90a237f2613935c4c7ccd79ef124fa73
Nov 02 19:53:54.311882 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:54.311863    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2hhpv"]
Nov 02 19:53:54.614888 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:54.614793    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2hhpv" event={"ID":"27c5829b-2a4f-4ddd-b413-c8d8268ff8c1","Type":"ContainerStarted","Data":"276b85a8e84611cd7986f93c59222b0b90a237f2613935c4c7ccd79ef124fa73"}
Nov 02 19:53:54.616265 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:54.616239    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8" event={"ID":"a3e59fe7-4223-4124-ba42-c5dde81b4c72","Type":"ContainerStarted","Data":"f76d1c37188ab65e1e40371f660a3298ddd86ffe435c99a5b9e2d240d4ceee03"}
Nov 02 19:53:54.733031 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:54.732982    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-g4db8" podStartSLOduration=2.6552014550000003 podStartE2EDuration="5.732966684s" totalImagesPullingTime="3.077765229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:54.731497323 +0000 UTC m=+315.729185932" watchObservedRunningTime="2026-11-02 19:53:54.732966684 +0000 UTC m=+315.730655293"
Nov 02 19:53:55.076940 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:55.076906    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:55.300283 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:55.300252    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:56.289083 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.289040    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 19:53:56.293926 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.293903    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"
Nov 02 19:53:56.350827 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.350798    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 19:53:56.350827 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.350798    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d2pr6\""
Nov 02 19:53:56.403403 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.403376    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f554e67-b005-4058-9728-e047ee0f014b-config-file\") pod \"limitador-limitador-7c9b87d574-2ptm5\" (UID: \"1f554e67-b005-4058-9728-e047ee0f014b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"
Nov 02 19:53:56.403403 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.403387    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 19:53:56.403605 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.403427    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt7g6\" (UniqueName: \"kubernetes.io/projected/1f554e67-b005-4058-9728-e047ee0f014b-kube-api-access-nt7g6\") pod \"limitador-limitador-7c9b87d574-2ptm5\" (UID: \"1f554e67-b005-4058-9728-e047ee0f014b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"
Nov 02 19:53:56.504311 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.504274    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f554e67-b005-4058-9728-e047ee0f014b-config-file\") pod \"limitador-limitador-7c9b87d574-2ptm5\" (UID: \"1f554e67-b005-4058-9728-e047ee0f014b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"
Nov 02 19:53:56.504488 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.504322    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7g6\" (UniqueName: \"kubernetes.io/projected/1f554e67-b005-4058-9728-e047ee0f014b-kube-api-access-nt7g6\") pod \"limitador-limitador-7c9b87d574-2ptm5\" (UID: \"1f554e67-b005-4058-9728-e047ee0f014b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"
Nov 02 19:53:56.504926 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.504905    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f554e67-b005-4058-9728-e047ee0f014b-config-file\") pod \"limitador-limitador-7c9b87d574-2ptm5\" (UID: \"1f554e67-b005-4058-9728-e047ee0f014b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"
Nov 02 19:53:56.522268 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.522236    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:56.562557 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.562475    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7g6\" (UniqueName: \"kubernetes.io/projected/1f554e67-b005-4058-9728-e047ee0f014b-kube-api-access-nt7g6\") pod \"limitador-limitador-7c9b87d574-2ptm5\" (UID: \"1f554e67-b005-4058-9728-e047ee0f014b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"
Nov 02 19:53:56.606057 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.606021    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"
Nov 02 19:53:56.607516 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.607495    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 19:53:56.706214 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.706173    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:56.971548 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:53:56.971516    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f554e67_b005_4058_9728_e047ee0f014b.slice/crio-2597ccb1b18d7dbc7c0ded00c65081ec39cde9b609fbf10a794f448a3c593e14 WatchSource:0}: Error finding container 2597ccb1b18d7dbc7c0ded00c65081ec39cde9b609fbf10a794f448a3c593e14: Status 404 returned error can't find the container with id 2597ccb1b18d7dbc7c0ded00c65081ec39cde9b609fbf10a794f448a3c593e14
Nov 02 19:53:56.973968 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:56.973946    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 19:53:57.331582 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:57.331541    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:57.356414 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:53:57.356381    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:53:57.632020 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:57.631931    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2ptm5" event={"ID":"1f554e67-b005-4058-9728-e047ee0f014b","Type":"ContainerStarted","Data":"2597ccb1b18d7dbc7c0ded00c65081ec39cde9b609fbf10a794f448a3c593e14"}
Nov 02 19:53:58.954083 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:58.954045    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:53:59.174875 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:53:59.174843    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:00.646181 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:00.646141    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2ptm5" event={"ID":"1f554e67-b005-4058-9728-e047ee0f014b","Type":"ContainerStarted","Data":"2899bc80e458470ab52a8429789f3d7b2ead1fec198b8a1f0ba9924aed1d857f"}
Nov 02 19:54:00.646559 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:00.646254    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"
Nov 02 19:54:00.867696 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:00.867642    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-2ptm5" podStartSLOduration=2.033048447 podStartE2EDuration="4.867630046s" totalImagesPullingTime="2.834581599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:00.866373006 +0000 UTC m=+321.864061640" watchObservedRunningTime="2026-11-02 19:54:00.867630046 +0000 UTC m=+321.865318654"
Nov 02 19:54:01.395315 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:01.395285    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2hhpv"]
Nov 02 19:54:03.661826 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:03.661788    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2hhpv" event={"ID":"27c5829b-2a4f-4ddd-b413-c8d8268ff8c1","Type":"ContainerStarted","Data":"9dc4a5c6acd8380dbf61c6dad99f371b62a221cf4aa9f75b49028cb609eb5890"}
Nov 02 19:54:03.662241 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:03.661794    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-2hhpv" podUID="27c5829b-2a4f-4ddd-b413-c8d8268ff8c1" containerName="authorino" containerID="cri-o://9dc4a5c6acd8380dbf61c6dad99f371b62a221cf4aa9f75b49028cb609eb5890" gracePeriod=30
Nov 02 19:54:03.777870 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:03.777821    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-2hhpv" podStartSLOduration=2.460135524 podStartE2EDuration="10.777805806s" totalImagesPullingTime="8.317670282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:03.776741411 +0000 UTC m=+324.774430021" watchObservedRunningTime="2026-11-02 19:54:03.777805806 +0000 UTC m=+324.775494415"
Nov 02 19:54:03.910732 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:03.910699    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2hhpv"
Nov 02 19:54:03.970462 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:03.970374    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:04.071797 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:04.071750    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c5829b-2a4f-4ddd-b413-c8d8268ff8c1-kube-api-access-89kp5\" (UniqueName: \"kubernetes.io/projected/27c5829b-2a4f-4ddd-b413-c8d8268ff8c1-kube-api-access-89kp5\") pod \"27c5829b-2a4f-4ddd-b413-c8d8268ff8c1\" (UID: \"27c5829b-2a4f-4ddd-b413-c8d8268ff8c1\") "
Nov 02 19:54:04.073919 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:04.073894    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c5829b-2a4f-4ddd-b413-c8d8268ff8c1-kube-api-access-89kp5" pod "27c5829b-2a4f-4ddd-b413-c8d8268ff8c1" (UID: "27c5829b-2a4f-4ddd-b413-c8d8268ff8c1"). InnerVolumeSpecName "kube-api-access-89kp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:54:04.173180 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:04.173137    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89kp5\" (UniqueName: \"kubernetes.io/projected/27c5829b-2a4f-4ddd-b413-c8d8268ff8c1-kube-api-access-89kp5\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:54:04.228291 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:04.228200    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:04.667310 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:04.667276    5224 generic.go:356] "Generic (PLEG): container finished" podID="27c5829b-2a4f-4ddd-b413-c8d8268ff8c1" containerID="9dc4a5c6acd8380dbf61c6dad99f371b62a221cf4aa9f75b49028cb609eb5890" exitCode=0
Nov 02 19:54:04.667725 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:04.667321    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2hhpv"
Nov 02 19:54:04.667725 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:04.667361    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2hhpv" event={"ID":"27c5829b-2a4f-4ddd-b413-c8d8268ff8c1","Type":"ContainerDied","Data":"9dc4a5c6acd8380dbf61c6dad99f371b62a221cf4aa9f75b49028cb609eb5890"}
Nov 02 19:54:04.667725 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:04.667400    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2hhpv" event={"ID":"27c5829b-2a4f-4ddd-b413-c8d8268ff8c1","Type":"ContainerDied","Data":"276b85a8e84611cd7986f93c59222b0b90a237f2613935c4c7ccd79ef124fa73"}
Nov 02 19:54:04.667725 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:04.667417    5224 scope.go:122] "RemoveContainer" containerID="9dc4a5c6acd8380dbf61c6dad99f371b62a221cf4aa9f75b49028cb609eb5890"
Nov 02 19:54:04.676067 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:04.676051    5224 scope.go:122] "RemoveContainer" containerID="9dc4a5c6acd8380dbf61c6dad99f371b62a221cf4aa9f75b49028cb609eb5890"
Nov 02 19:54:04.676324 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:54:04.676306    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc4a5c6acd8380dbf61c6dad99f371b62a221cf4aa9f75b49028cb609eb5890\": container with ID starting with 9dc4a5c6acd8380dbf61c6dad99f371b62a221cf4aa9f75b49028cb609eb5890 not found: ID does not exist" containerID="9dc4a5c6acd8380dbf61c6dad99f371b62a221cf4aa9f75b49028cb609eb5890"
Nov 02 19:54:04.676364 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:04.676333    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc4a5c6acd8380dbf61c6dad99f371b62a221cf4aa9f75b49028cb609eb5890"} err="failed to get container status \"9dc4a5c6acd8380dbf61c6dad99f371b62a221cf4aa9f75b49028cb609eb5890\": rpc error: code = NotFound desc = could not find container \"9dc4a5c6acd8380dbf61c6dad99f371b62a221cf4aa9f75b49028cb609eb5890\": container with ID starting with 9dc4a5c6acd8380dbf61c6dad99f371b62a221cf4aa9f75b49028cb609eb5890 not found: ID does not exist"
Nov 02 19:54:04.832483 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:04.832450    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2hhpv"]
Nov 02 19:54:04.836473 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:04.836453    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2hhpv"]
Nov 02 19:54:05.165313 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:05.165281    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c5829b-2a4f-4ddd-b413-c8d8268ff8c1" path="/var/lib/kubelet/pods/27c5829b-2a4f-4ddd-b413-c8d8268ff8c1/volumes"
Nov 02 19:54:05.718268 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:05.718227    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:05.895964 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:05.895924    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:06.106583 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:06.106537    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:06.282761 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:06.282732    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:06.566087 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:06.566039    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:06.748978 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:06.748940    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:10.991649 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:10.991611    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:11.162551 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:11.162519    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:11.413074 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:11.413042    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:11.652306 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:11.652277    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"
Nov 02 19:54:13.521781 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:13.521748    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:13.690972 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:13.690926    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:18.346044 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:18.346001    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:18.584467 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:18.584422    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:20.460747 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:20.460697    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:20.665181 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:20.665147    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:20.841213 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:20.841181    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:21.259249 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:21.259156    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:27.357632 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:54:27.357595    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:54:40.173320 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:40.173283    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:41.678364 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:41.678325    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:41.902399 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:41.902366    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:42.076693 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:42.076658    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:46.686985 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:46.686946    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:47.441332 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:47.441294    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:51.954180 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:51.954093    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:52.173762 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:52.173722    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:54.048090 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:54.048051    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:54.261341 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:54:54.261308    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:54:57.357782 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:54:57.357730    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:55:20.074739 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.074699    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:20.286094 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.286043    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:20.560039 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.559983    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:20.736572 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.736494    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:20.886840 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.886742    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2vzvt"]
Nov 02 19:55:20.887142 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.887128    5224 cpu_manager.go:414] "containerMap: removing container" podUID="27c5829b-2a4f-4ddd-b413-c8d8268ff8c1" containerName="authorino"
Nov 02 19:55:20.887189 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.887143    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c5829b-2a4f-4ddd-b413-c8d8268ff8c1" containerName="authorino"
Nov 02 19:55:20.887236 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.887227    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c5829b-2a4f-4ddd-b413-c8d8268ff8c1" containerName="authorino"
Nov 02 19:55:20.890119 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.890103    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-2vzvt"
Nov 02 19:55:20.910693 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.910626    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:20.928900 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.928867    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb2304f9-c44a-44a4-a0b5-6029ffd68eec-cert\") pod \"kserve-controller-manager-776cc66d47-2vzvt\" (UID: \"cb2304f9-c44a-44a4-a0b5-6029ffd68eec\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2vzvt"
Nov 02 19:55:20.929113 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.929023    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47nfr\" (UniqueName: \"kubernetes.io/projected/cb2304f9-c44a-44a4-a0b5-6029ffd68eec-kube-api-access-47nfr\") pod \"kserve-controller-manager-776cc66d47-2vzvt\" (UID: \"cb2304f9-c44a-44a4-a0b5-6029ffd68eec\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2vzvt"
Nov 02 19:55:20.943918 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.943878    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2vzvt"]
Nov 02 19:55:20.979179 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.979146    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 19:55:20.993239 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.993207    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 19:55:20.993239 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.993207    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 02 19:55:20.993458 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.993251    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xrmxn\""
Nov 02 19:55:20.996331 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.996311    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"]
Nov 02 19:55:20.999582 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:20.999566    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"
Nov 02 19:55:21.029729 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.029690    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flwrq\" (UniqueName: \"kubernetes.io/projected/edc475f0-1afd-4845-88be-af3a13c65009-kube-api-access-flwrq\") pod \"llmisvc-controller-manager-869d5dcc5d-zdt9p\" (UID: \"edc475f0-1afd-4845-88be-af3a13c65009\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"
Nov 02 19:55:21.029966 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.029764    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nfr\" (UniqueName: \"kubernetes.io/projected/cb2304f9-c44a-44a4-a0b5-6029ffd68eec-kube-api-access-47nfr\") pod \"kserve-controller-manager-776cc66d47-2vzvt\" (UID: \"cb2304f9-c44a-44a4-a0b5-6029ffd68eec\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2vzvt"
Nov 02 19:55:21.029966 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.029865    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edc475f0-1afd-4845-88be-af3a13c65009-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-zdt9p\" (UID: \"edc475f0-1afd-4845-88be-af3a13c65009\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"
Nov 02 19:55:21.029966 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.029918    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb2304f9-c44a-44a4-a0b5-6029ffd68eec-cert\") pod \"kserve-controller-manager-776cc66d47-2vzvt\" (UID: \"cb2304f9-c44a-44a4-a0b5-6029ffd68eec\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2vzvt"
Nov 02 19:55:21.032402 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.032374    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb2304f9-c44a-44a4-a0b5-6029ffd68eec-cert\") pod \"kserve-controller-manager-776cc66d47-2vzvt\" (UID: \"cb2304f9-c44a-44a4-a0b5-6029ffd68eec\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2vzvt"
Nov 02 19:55:21.047376 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.047342    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"]
Nov 02 19:55:21.068512 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.068477    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mgcpp\""
Nov 02 19:55:21.068512 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.068511    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 02 19:55:21.090453 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.090416    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47nfr\" (UniqueName: \"kubernetes.io/projected/cb2304f9-c44a-44a4-a0b5-6029ffd68eec-kube-api-access-47nfr\") pod \"kserve-controller-manager-776cc66d47-2vzvt\" (UID: \"cb2304f9-c44a-44a4-a0b5-6029ffd68eec\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2vzvt"
Nov 02 19:55:21.100489 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.100450    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:21.131151 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.131107    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flwrq\" (UniqueName: \"kubernetes.io/projected/edc475f0-1afd-4845-88be-af3a13c65009-kube-api-access-flwrq\") pod \"llmisvc-controller-manager-869d5dcc5d-zdt9p\" (UID: \"edc475f0-1afd-4845-88be-af3a13c65009\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"
Nov 02 19:55:21.131340 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.131186    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edc475f0-1afd-4845-88be-af3a13c65009-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-zdt9p\" (UID: \"edc475f0-1afd-4845-88be-af3a13c65009\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"
Nov 02 19:55:21.133521 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.133501    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edc475f0-1afd-4845-88be-af3a13c65009-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-zdt9p\" (UID: \"edc475f0-1afd-4845-88be-af3a13c65009\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"
Nov 02 19:55:21.191207 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.191136    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flwrq\" (UniqueName: \"kubernetes.io/projected/edc475f0-1afd-4845-88be-af3a13c65009-kube-api-access-flwrq\") pod \"llmisvc-controller-manager-869d5dcc5d-zdt9p\" (UID: \"edc475f0-1afd-4845-88be-af3a13c65009\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"
Nov 02 19:55:21.202969 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.202935    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-2vzvt"
Nov 02 19:55:21.276613 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.276577    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:21.312435 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.312387    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"
Nov 02 19:55:21.577196 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:55:21.577165    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2304f9_c44a_44a4_a0b5_6029ffd68eec.slice/crio-3483039651d32d79fd14812e45f7a1b36e97dd1feb23790f1371851faa1f245f WatchSource:0}: Error finding container 3483039651d32d79fd14812e45f7a1b36e97dd1feb23790f1371851faa1f245f: Status 404 returned error can't find the container with id 3483039651d32d79fd14812e45f7a1b36e97dd1feb23790f1371851faa1f245f
Nov 02 19:55:21.579598 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.579574    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2vzvt"]
Nov 02 19:55:21.605620 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.605588    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:21.682471 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.682367    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"]
Nov 02 19:55:21.684077 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:55:21.684028    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podedc475f0_1afd_4845_88be_af3a13c65009.slice/crio-f4a2412cf37328911b17c108bc23fc24f6b1eb35018e545cd6b1f96288c488f4 WatchSource:0}: Error finding container f4a2412cf37328911b17c108bc23fc24f6b1eb35018e545cd6b1f96288c488f4: Status 404 returned error can't find the container with id f4a2412cf37328911b17c108bc23fc24f6b1eb35018e545cd6b1f96288c488f4
Nov 02 19:55:21.787083 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:21.787042    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:22.012428 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:22.012334    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-2vzvt" event={"ID":"cb2304f9-c44a-44a4-a0b5-6029ffd68eec","Type":"ContainerStarted","Data":"3483039651d32d79fd14812e45f7a1b36e97dd1feb23790f1371851faa1f245f"}
Nov 02 19:55:22.013524 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:22.013493    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p" event={"ID":"edc475f0-1afd-4845-88be-af3a13c65009","Type":"ContainerStarted","Data":"f4a2412cf37328911b17c108bc23fc24f6b1eb35018e545cd6b1f96288c488f4"}
Nov 02 19:55:22.235086 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:22.235031    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:23.809052 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:23.809012    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:24.028242 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:24.028120    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:24.203508 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:24.203477    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:25.033533 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:25.033494    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-2vzvt" event={"ID":"cb2304f9-c44a-44a4-a0b5-6029ffd68eec","Type":"ContainerStarted","Data":"e6463428abfff7be8b51443788fc3f7b709042870677dd49e3ceb2f6cfc84897"}
Nov 02 19:55:25.034044 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:25.033568    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-2vzvt"
Nov 02 19:55:25.035149 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:25.035125    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p" event={"ID":"edc475f0-1afd-4845-88be-af3a13c65009","Type":"ContainerStarted","Data":"7408feca6750cbea2bee34a624008de5f68200378fcc19fb24a9385452015647"}
Nov 02 19:55:25.035250 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:25.035172    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"
Nov 02 19:55:25.151209 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:25.151150    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-2vzvt" podStartSLOduration=2.475619382 podStartE2EDuration="5.151136922s" totalImagesPullingTime="2.67551754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:25.150175659 +0000 UTC m=+406.147864267" watchObservedRunningTime="2026-11-02 19:55:25.151136922 +0000 UTC m=+406.148825552"
Nov 02 19:55:25.266403 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:25.266344    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p" podStartSLOduration=2.773724321 podStartE2EDuration="5.266323575s" totalImagesPullingTime="2.492599254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:25.264989164 +0000 UTC m=+406.262677773" watchObservedRunningTime="2026-11-02 19:55:25.266323575 +0000 UTC m=+406.264012186"
Nov 02 19:55:27.034045 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:27.034007    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:27.203082 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:27.203046    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:27.361743 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:55:27.361657    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:55:27.374349 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:27.374317    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:27.556658 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:27.556627    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:27.761332 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:27.761237    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:27.936601 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:27.936551    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:28.107596 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:28.107555    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:28.278831 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:28.278791    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:32.139333 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:32.139291    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 19:55:39.733130 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:39.733089    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:40.058123 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:40.058083    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:40.367501 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:40.367417    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b6977d64c-jlddm"]
Nov 02 19:55:40.370808 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:40.370780    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b6977d64c-jlddm"
Nov 02 19:55:40.414900 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:40.414868    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b6977d64c-jlddm"]
Nov 02 19:55:40.464598 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:40.464567    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vvsx7\""
Nov 02 19:55:40.506204 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:40.506162    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbc7h\" (UniqueName: \"kubernetes.io/projected/a8b9d431-2ed3-41d5-b131-9f1f07c40e48-kube-api-access-lbc7h\") pod \"authorino-5b6977d64c-jlddm\" (UID: \"a8b9d431-2ed3-41d5-b131-9f1f07c40e48\") " pod="kuadrant-system/authorino-5b6977d64c-jlddm"
Nov 02 19:55:40.607085 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:40.607038    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbc7h\" (UniqueName: \"kubernetes.io/projected/a8b9d431-2ed3-41d5-b131-9f1f07c40e48-kube-api-access-lbc7h\") pod \"authorino-5b6977d64c-jlddm\" (UID: \"a8b9d431-2ed3-41d5-b131-9f1f07c40e48\") " pod="kuadrant-system/authorino-5b6977d64c-jlddm"
Nov 02 19:55:40.664918 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:40.664854    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbc7h\" (UniqueName: \"kubernetes.io/projected/a8b9d431-2ed3-41d5-b131-9f1f07c40e48-kube-api-access-lbc7h\") pod \"authorino-5b6977d64c-jlddm\" (UID: \"a8b9d431-2ed3-41d5-b131-9f1f07c40e48\") " pod="kuadrant-system/authorino-5b6977d64c-jlddm"
Nov 02 19:55:40.682693 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:40.682658    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b6977d64c-jlddm"
Nov 02 19:55:40.796321 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:40.796286    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b6977d64c-jlddm"]
Nov 02 19:55:41.048843 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:55:41.048815    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8b9d431_2ed3_41d5_b131_9f1f07c40e48.slice/crio-a08fb8d553ce14997fd19096df417c368307fa3a372cd47dba16a835a9668b02 WatchSource:0}: Error finding container a08fb8d553ce14997fd19096df417c368307fa3a372cd47dba16a835a9668b02: Status 404 returned error can't find the container with id a08fb8d553ce14997fd19096df417c368307fa3a372cd47dba16a835a9668b02
Nov 02 19:55:41.051190 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:41.051163    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b6977d64c-jlddm"]
Nov 02 19:55:41.120411 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:41.120372    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b6977d64c-jlddm" event={"ID":"a8b9d431-2ed3-41d5-b131-9f1f07c40e48","Type":"ContainerStarted","Data":"a08fb8d553ce14997fd19096df417c368307fa3a372cd47dba16a835a9668b02"}
Nov 02 19:55:43.132019 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:43.131984    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b6977d64c-jlddm" event={"ID":"a8b9d431-2ed3-41d5-b131-9f1f07c40e48","Type":"ContainerStarted","Data":"850580b9d061e5b6b080d71d7b4adc7d5106f7a19fe815ddeaf85d216ece522a"}
Nov 02 19:55:43.132450 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:43.132072    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5b6977d64c-jlddm" podUID="a8b9d431-2ed3-41d5-b131-9f1f07c40e48" containerName="authorino" containerID="cri-o://850580b9d061e5b6b080d71d7b4adc7d5106f7a19fe815ddeaf85d216ece522a" gracePeriod=30
Nov 02 19:55:43.385214 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:43.385146    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b6977d64c-jlddm"
Nov 02 19:55:43.536409 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:43.536368    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b9d431-2ed3-41d5-b131-9f1f07c40e48-kube-api-access-lbc7h\" (UniqueName: \"kubernetes.io/projected/a8b9d431-2ed3-41d5-b131-9f1f07c40e48-kube-api-access-lbc7h\") pod \"a8b9d431-2ed3-41d5-b131-9f1f07c40e48\" (UID: \"a8b9d431-2ed3-41d5-b131-9f1f07c40e48\") "
Nov 02 19:55:43.538469 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:43.538441    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b9d431-2ed3-41d5-b131-9f1f07c40e48-kube-api-access-lbc7h" pod "a8b9d431-2ed3-41d5-b131-9f1f07c40e48" (UID: "a8b9d431-2ed3-41d5-b131-9f1f07c40e48"). InnerVolumeSpecName "kube-api-access-lbc7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:55:43.637728 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:43.637636    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbc7h\" (UniqueName: \"kubernetes.io/projected/a8b9d431-2ed3-41d5-b131-9f1f07c40e48-kube-api-access-lbc7h\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:55:44.138502 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:44.138464    5224 generic.go:356] "Generic (PLEG): container finished" podID="a8b9d431-2ed3-41d5-b131-9f1f07c40e48" containerID="850580b9d061e5b6b080d71d7b4adc7d5106f7a19fe815ddeaf85d216ece522a" exitCode=0
Nov 02 19:55:44.139014 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:44.138511    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b6977d64c-jlddm"
Nov 02 19:55:44.139014 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:44.138550    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b6977d64c-jlddm" event={"ID":"a8b9d431-2ed3-41d5-b131-9f1f07c40e48","Type":"ContainerDied","Data":"850580b9d061e5b6b080d71d7b4adc7d5106f7a19fe815ddeaf85d216ece522a"}
Nov 02 19:55:44.139014 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:44.138595    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b6977d64c-jlddm" event={"ID":"a8b9d431-2ed3-41d5-b131-9f1f07c40e48","Type":"ContainerDied","Data":"a08fb8d553ce14997fd19096df417c368307fa3a372cd47dba16a835a9668b02"}
Nov 02 19:55:44.139014 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:44.138618    5224 scope.go:122] "RemoveContainer" containerID="850580b9d061e5b6b080d71d7b4adc7d5106f7a19fe815ddeaf85d216ece522a"
Nov 02 19:55:44.154904 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:44.154879    5224 scope.go:122] "RemoveContainer" containerID="850580b9d061e5b6b080d71d7b4adc7d5106f7a19fe815ddeaf85d216ece522a"
Nov 02 19:55:44.155203 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:55:44.155183    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850580b9d061e5b6b080d71d7b4adc7d5106f7a19fe815ddeaf85d216ece522a\": container with ID starting with 850580b9d061e5b6b080d71d7b4adc7d5106f7a19fe815ddeaf85d216ece522a not found: ID does not exist" containerID="850580b9d061e5b6b080d71d7b4adc7d5106f7a19fe815ddeaf85d216ece522a"
Nov 02 19:55:44.155270 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:44.155213    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850580b9d061e5b6b080d71d7b4adc7d5106f7a19fe815ddeaf85d216ece522a"} err="failed to get container status \"850580b9d061e5b6b080d71d7b4adc7d5106f7a19fe815ddeaf85d216ece522a\": rpc error: code = NotFound desc = could not find container \"850580b9d061e5b6b080d71d7b4adc7d5106f7a19fe815ddeaf85d216ece522a\": container with ID starting with 850580b9d061e5b6b080d71d7b4adc7d5106f7a19fe815ddeaf85d216ece522a not found: ID does not exist"
Nov 02 19:55:44.304264 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:44.304225    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b6977d64c-jlddm"]
Nov 02 19:55:44.306287 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:44.306261    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b6977d64c-jlddm"]
Nov 02 19:55:45.165724 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:45.165690    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b9d431-2ed3-41d5-b131-9f1f07c40e48" path="/var/lib/kubelet/pods/a8b9d431-2ed3-41d5-b131-9f1f07c40e48/volumes"
Nov 02 19:55:47.016166 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:47.016131    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:47.739443 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:47.739408    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:55:56.042193 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:56.042155    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"
Nov 02 19:55:56.045102 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:55:56.045081    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-2vzvt"
Nov 02 19:55:57.359595 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:55:57.359561    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:56:20.899328 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:20.899240    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:56:25.348362 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:25.348327    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw"]
Nov 02 19:56:25.348745 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:25.348683    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a8b9d431-2ed3-41d5-b131-9f1f07c40e48" containerName="authorino"
Nov 02 19:56:25.348745 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:25.348694    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b9d431-2ed3-41d5-b131-9f1f07c40e48" containerName="authorino"
Nov 02 19:56:25.348847 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:25.348797    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b9d431-2ed3-41d5-b131-9f1f07c40e48" containerName="authorino"
Nov 02 19:56:25.350789 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:25.350759    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw"
Nov 02 19:56:25.388604 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:25.388567    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e49ef931-b6e2-4394-80f3-881252323f43-cert\") pod \"llmisvc-controller-manager-6466c554d8-8z8gw\" (UID: \"e49ef931-b6e2-4394-80f3-881252323f43\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw"
Nov 02 19:56:25.388792 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:25.388710    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb8th\" (UniqueName: \"kubernetes.io/projected/e49ef931-b6e2-4394-80f3-881252323f43-kube-api-access-sb8th\") pod \"llmisvc-controller-manager-6466c554d8-8z8gw\" (UID: \"e49ef931-b6e2-4394-80f3-881252323f43\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw"
Nov 02 19:56:25.404712 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:25.404681    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw"]
Nov 02 19:56:25.489795 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:25.489727    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb8th\" (UniqueName: \"kubernetes.io/projected/e49ef931-b6e2-4394-80f3-881252323f43-kube-api-access-sb8th\") pod \"llmisvc-controller-manager-6466c554d8-8z8gw\" (UID: \"e49ef931-b6e2-4394-80f3-881252323f43\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw"
Nov 02 19:56:25.490009 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:25.489807    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e49ef931-b6e2-4394-80f3-881252323f43-cert\") pod \"llmisvc-controller-manager-6466c554d8-8z8gw\" (UID: \"e49ef931-b6e2-4394-80f3-881252323f43\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw"
Nov 02 19:56:25.492234 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:25.492211    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e49ef931-b6e2-4394-80f3-881252323f43-cert\") pod \"llmisvc-controller-manager-6466c554d8-8z8gw\" (UID: \"e49ef931-b6e2-4394-80f3-881252323f43\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw"
Nov 02 19:56:25.548051 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:25.548015    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb8th\" (UniqueName: \"kubernetes.io/projected/e49ef931-b6e2-4394-80f3-881252323f43-kube-api-access-sb8th\") pod \"llmisvc-controller-manager-6466c554d8-8z8gw\" (UID: \"e49ef931-b6e2-4394-80f3-881252323f43\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw"
Nov 02 19:56:25.662132 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:25.662042    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw"
Nov 02 19:56:26.026865 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:56:26.026834    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49ef931_b6e2_4394_80f3_881252323f43.slice/crio-ddabf8c25032848705fb14241a24050b9d700d963d649380421483304a915ecc WatchSource:0}: Error finding container ddabf8c25032848705fb14241a24050b9d700d963d649380421483304a915ecc: Status 404 returned error can't find the container with id ddabf8c25032848705fb14241a24050b9d700d963d649380421483304a915ecc
Nov 02 19:56:26.029048 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:26.029018    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw"]
Nov 02 19:56:26.333161 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:26.333073    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw" event={"ID":"e49ef931-b6e2-4394-80f3-881252323f43","Type":"ContainerStarted","Data":"ddabf8c25032848705fb14241a24050b9d700d963d649380421483304a915ecc"}
Nov 02 19:56:27.338540 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:27.338502    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw" event={"ID":"e49ef931-b6e2-4394-80f3-881252323f43","Type":"ContainerStarted","Data":"f5f961ac0ad86e1de9d6a769e4a0e86e2bc5b246f30ac3167ebe72227c57651a"}
Nov 02 19:56:27.339067 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:27.338567    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw"
Nov 02 19:56:27.358483 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:56:27.358448    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:56:27.454363 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:27.454308    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw" podStartSLOduration=1.678367537 podStartE2EDuration="2.454294704s" totalImagesPullingTime="775.927167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:27.454072664 +0000 UTC m=+468.451761285" watchObservedRunningTime="2026-11-02 19:56:27.454294704 +0000 UTC m=+468.451983313"
Nov 02 19:56:47.359481 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:47.359441    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:56:48.421850 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:48.421813    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:56:57.357482 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:56:57.357431    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:56:58.346299 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:58.346269    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8z8gw"
Nov 02 19:56:58.515264 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:58.515223    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"]
Nov 02 19:56:58.515669 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:58.515502    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p" podUID="edc475f0-1afd-4845-88be-af3a13c65009" containerName="manager" containerID="cri-o://7408feca6750cbea2bee34a624008de5f68200378fcc19fb24a9385452015647" gracePeriod=30
Nov 02 19:56:58.753901 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:58.753869    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"
Nov 02 19:56:58.883370 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:58.883290    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc475f0-1afd-4845-88be-af3a13c65009-kube-api-access-flwrq\" (UniqueName: \"kubernetes.io/projected/edc475f0-1afd-4845-88be-af3a13c65009-kube-api-access-flwrq\") pod \"edc475f0-1afd-4845-88be-af3a13c65009\" (UID: \"edc475f0-1afd-4845-88be-af3a13c65009\") "
Nov 02 19:56:58.883511 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:58.883374    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edc475f0-1afd-4845-88be-af3a13c65009-cert\" (UniqueName: \"kubernetes.io/secret/edc475f0-1afd-4845-88be-af3a13c65009-cert\") pod \"edc475f0-1afd-4845-88be-af3a13c65009\" (UID: \"edc475f0-1afd-4845-88be-af3a13c65009\") "
Nov 02 19:56:58.885529 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:58.885497    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc475f0-1afd-4845-88be-af3a13c65009-kube-api-access-flwrq" pod "edc475f0-1afd-4845-88be-af3a13c65009" (UID: "edc475f0-1afd-4845-88be-af3a13c65009"). InnerVolumeSpecName "kube-api-access-flwrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:56:58.885639 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:58.885499    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc475f0-1afd-4845-88be-af3a13c65009-cert" pod "edc475f0-1afd-4845-88be-af3a13c65009" (UID: "edc475f0-1afd-4845-88be-af3a13c65009"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:56:58.984536 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:58.984505    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flwrq\" (UniqueName: \"kubernetes.io/projected/edc475f0-1afd-4845-88be-af3a13c65009-kube-api-access-flwrq\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:56:58.984536 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:58.984533    5224 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edc475f0-1afd-4845-88be-af3a13c65009-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:56:59.488210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:59.488108    5224 generic.go:356] "Generic (PLEG): container finished" podID="edc475f0-1afd-4845-88be-af3a13c65009" containerID="7408feca6750cbea2bee34a624008de5f68200378fcc19fb24a9385452015647" exitCode=0
Nov 02 19:56:59.488210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:59.488174    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"
Nov 02 19:56:59.488210 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:59.488189    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p" event={"ID":"edc475f0-1afd-4845-88be-af3a13c65009","Type":"ContainerDied","Data":"7408feca6750cbea2bee34a624008de5f68200378fcc19fb24a9385452015647"}
Nov 02 19:56:59.488498 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:59.488252    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p" event={"ID":"edc475f0-1afd-4845-88be-af3a13c65009","Type":"ContainerDied","Data":"f4a2412cf37328911b17c108bc23fc24f6b1eb35018e545cd6b1f96288c488f4"}
Nov 02 19:56:59.488498 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:59.488275    5224 scope.go:122] "RemoveContainer" containerID="7408feca6750cbea2bee34a624008de5f68200378fcc19fb24a9385452015647"
Nov 02 19:56:59.496894 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:59.496835    5224 scope.go:122] "RemoveContainer" containerID="7408feca6750cbea2bee34a624008de5f68200378fcc19fb24a9385452015647"
Nov 02 19:56:59.497173 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:56:59.497150    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7408feca6750cbea2bee34a624008de5f68200378fcc19fb24a9385452015647\": container with ID starting with 7408feca6750cbea2bee34a624008de5f68200378fcc19fb24a9385452015647 not found: ID does not exist" containerID="7408feca6750cbea2bee34a624008de5f68200378fcc19fb24a9385452015647"
Nov 02 19:56:59.497235 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:59.497183    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7408feca6750cbea2bee34a624008de5f68200378fcc19fb24a9385452015647"} err="failed to get container status \"7408feca6750cbea2bee34a624008de5f68200378fcc19fb24a9385452015647\": rpc error: code = NotFound desc = could not find container \"7408feca6750cbea2bee34a624008de5f68200378fcc19fb24a9385452015647\": container with ID starting with 7408feca6750cbea2bee34a624008de5f68200378fcc19fb24a9385452015647 not found: ID does not exist"
Nov 02 19:56:59.655756 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:59.655720    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"]
Nov 02 19:56:59.659621 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:59.659600    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zdt9p"]
Nov 02 19:56:59.686915 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:59.686881    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:56:59.923347 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:56:59.923310    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:01.165953 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:01.165915    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc475f0-1afd-4845-88be-af3a13c65009" path="/var/lib/kubelet/pods/edc475f0-1afd-4845-88be-af3a13c65009/volumes"
Nov 02 19:57:09.008239 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:09.008199    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:09.194593 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:09.194543    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:24.299813 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:24.299776    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:27.357093 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:57:27.357060    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:57:31.372057 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:31.372017    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:34.119542 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:34.119506    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:35.624846 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:35.624814    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:36.575003 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:36.574964    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:36.764178 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:36.764143    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:40.618333 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:40.618297    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:43.904109 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:43.904077    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 19:57:47.700313 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:47.700270    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:48.390360 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:48.390273    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:48.637277 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:48.637234    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:51.271480 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:51.271446    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 19:57:52.553783 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:52.553741    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:56.112518 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:56.112481    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:57:57.357638 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:57:57.357600    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:57:59.996942 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:57:59.996904    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:58:00.378677 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:00.378642    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 19:58:03.453137 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:03.453101    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:58:06.892358 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:06.892326    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"]
Nov 02 19:58:06.892737 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:06.892663    5224 cpu_manager.go:414] "containerMap: removing container" podUID="edc475f0-1afd-4845-88be-af3a13c65009" containerName="manager"
Nov 02 19:58:06.892737 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:06.892673    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc475f0-1afd-4845-88be-af3a13c65009" containerName="manager"
Nov 02 19:58:06.892737 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:06.892737    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc475f0-1afd-4845-88be-af3a13c65009" containerName="manager"
Nov 02 19:58:06.895798 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:06.895783    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:06.949336 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:06.949299    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-phngj\""
Nov 02 19:58:07.002105 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.002074    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"]
Nov 02 19:58:07.048799 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.048744    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.048975 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.048811    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bd828b-4265-447e-89da-c94e97503b89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.048975 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.048902    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.048975 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.048961    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.049082 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.049015    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.049082 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.049058    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.049082 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.049075    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bd828b-4265-447e-89da-c94e97503b89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.049198 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.049090    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdnlb\" (UniqueName: \"kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-kube-api-access-qdnlb\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.049238 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.049190    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.150392 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.150306    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.150392 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.150378    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.150621 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.150413    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.150621 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.150476    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.150621 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.150504    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bd828b-4265-447e-89da-c94e97503b89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.150621 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.150528    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdnlb\" (UniqueName: \"kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-kube-api-access-qdnlb\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.150621 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.150594    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.150910 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.150625    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.150910 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.150661    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bd828b-4265-447e-89da-c94e97503b89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.150910 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.150807    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.150910 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.150807    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.150910 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.150897    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.151172 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.151064    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.151398 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.151374    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bd828b-4265-447e-89da-c94e97503b89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.152872 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.152845    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.152963 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.152919    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bd828b-4265-447e-89da-c94e97503b89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.209242 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.209212    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.209427 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.209407    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdnlb\" (UniqueName: \"kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-kube-api-access-qdnlb\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9n4hv\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.265121 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.265073    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:58:07.509040 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.508963    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:07.836608 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.836575    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:58:07.919890 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:58:07.919857    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bd828b_4265_447e_89da_c94e97503b89.slice/crio-ea62bbb199734f03ce30cc935ebe26c471cf822822245da61f59f96454b637d9 WatchSource:0}: Error finding container ea62bbb199734f03ce30cc935ebe26c471cf822822245da61f59f96454b637d9: Status 404 returned error can't find the container with id ea62bbb199734f03ce30cc935ebe26c471cf822822245da61f59f96454b637d9
Nov 02 19:58:07.922125 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.922087    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 19:58:07.922221 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.922173    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 19:58:07.922221 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.922214    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 19:58:07.923293 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:07.923274    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"]
Nov 02 19:58:08.010151 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:08.010122    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:58:08.800839 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:08.800762    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" event={"ID":"a9bd828b-4265-447e-89da-c94e97503b89","Type":"ContainerStarted","Data":"875af3bd659c015bd90814f346426da7b1f1aace5830778b9dde91fabe57df24"}
Nov 02 19:58:08.800839 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:08.800841    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" event={"ID":"a9bd828b-4265-447e-89da-c94e97503b89","Type":"ContainerStarted","Data":"ea62bbb199734f03ce30cc935ebe26c471cf822822245da61f59f96454b637d9"}
Nov 02 19:58:09.027901 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:09.027850    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" podStartSLOduration=3.027834222 podStartE2EDuration="3.027834222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:08.976318577 +0000 UTC m=+569.974007187" watchObservedRunningTime="2026-11-02 19:58:09.027834222 +0000 UTC m=+570.025522831"
Nov 02 19:58:09.509999 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:09.509967    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:09.511416 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:09.511386    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 19:58:09.511548 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:09.511442    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" podUID="a9bd828b-4265-447e-89da-c94e97503b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 19:58:10.509970 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:10.509925    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 19:58:10.510363 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:10.509992    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" podUID="a9bd828b-4265-447e-89da-c94e97503b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 19:58:11.068792 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:11.068078    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:58:11.509661 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:11.509582    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 19:58:11.509661 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:11.509631    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" podUID="a9bd828b-4265-447e-89da-c94e97503b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 19:58:12.510209 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:12.510173    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 19:58:12.510602 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:12.510227    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" podUID="a9bd828b-4265-447e-89da-c94e97503b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 19:58:13.510325 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:13.510292    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 19:58:13.510711 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:13.510344    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" podUID="a9bd828b-4265-447e-89da-c94e97503b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 19:58:13.544578 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:13.544545    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 19:58:14.509873 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:14.509835    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 19:58:14.510038 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:14.509907    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" podUID="a9bd828b-4265-447e-89da-c94e97503b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 19:58:15.654215 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:15.654182    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:15.654640 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:15.654526    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:15.655464 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:15.655446    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 19:58:22.029587 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.029548    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"]
Nov 02 19:58:22.032887 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.032865    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.092253 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.092220    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.092429 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.092276    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.092429 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.092297    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.092429 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.092354    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.092669 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.092501    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.092669 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.092539    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.092669 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.092585    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfmgj\" (UniqueName: \"kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-kube-api-access-wfmgj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.092669 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.092631    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.092875 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.092732    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.140865 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.140838    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"]
Nov 02 19:58:22.193460 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.193434    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.193460 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.193464    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.193702 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.193483    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.193702 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.193540    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.193702 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.193564    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.193702 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.193586    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmgj\" (UniqueName: \"kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-kube-api-access-wfmgj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.193702 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.193629    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.194026 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.193711    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.194026 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.193749    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.194128 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.194085    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.194188 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.194147    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.194350 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.194325    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.194424 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.194372    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.194581 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.194558    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.195894 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.195875    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.196309 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.196290    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.251565 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.251536    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.251674 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.251616    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfmgj\" (UniqueName: \"kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-kube-api-access-wfmgj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4ms8z\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.345697 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.345624    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:22.762514 ip-10-0-134-196 kubenswrapper[5224]: W1102 19:58:22.762482    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c802b4b_1dec_400b_a1e7_5507933ef8e3.slice/crio-c6122f9f3cc39897b0f2e7fdb1be3ebe35ac66847fe08c9a0650f7d5e9c4335c WatchSource:0}: Error finding container c6122f9f3cc39897b0f2e7fdb1be3ebe35ac66847fe08c9a0650f7d5e9c4335c: Status 404 returned error can't find the container with id c6122f9f3cc39897b0f2e7fdb1be3ebe35ac66847fe08c9a0650f7d5e9c4335c
Nov 02 19:58:22.764321 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.764301    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"]
Nov 02 19:58:22.765264 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.765228    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 19:58:22.765344 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.765307    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 19:58:22.765389 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.765350    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 19:58:22.870122 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.870086    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" event={"ID":"8c802b4b-1dec-400b-a1e7-5507933ef8e3","Type":"ContainerStarted","Data":"d63cecc41ad759491c5e5533bb495d3f39518fc6ffbd5f2cab6e5dbe9822cd1e"}
Nov 02 19:58:22.870302 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:22.870144    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" event={"ID":"8c802b4b-1dec-400b-a1e7-5507933ef8e3","Type":"ContainerStarted","Data":"c6122f9f3cc39897b0f2e7fdb1be3ebe35ac66847fe08c9a0650f7d5e9c4335c"}
Nov 02 19:58:23.006373 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:23.006311    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" podStartSLOduration=2.006294513 podStartE2EDuration="2.006294513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:23.003788993 +0000 UTC m=+584.001477595" watchObservedRunningTime="2026-11-02 19:58:23.006294513 +0000 UTC m=+584.003983123"
Nov 02 19:58:23.345925 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:23.345889    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:23.347352 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:23.347332    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 19:58:23.347452 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:23.347378    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" podUID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 19:58:24.346545 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:24.346512    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 19:58:24.346948 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:24.346565    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" podUID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 19:58:25.346393 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:25.346357    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 19:58:25.346601 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:25.346417    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" podUID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 19:58:26.346827 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:26.346793    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 19:58:26.347344 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:26.346846    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" podUID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 19:58:27.346048 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:27.346012    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 19:58:27.346227 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:27.346060    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" podUID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 19:58:27.381646 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:58:27.381603    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:58:28.346952 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:28.346918    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 19:58:28.347208 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:28.347185    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" podUID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 19:58:29.346300 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:29.346267    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 19:58:29.346674 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:29.346317    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" podUID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 19:58:31.027155 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:31.027121    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:31.027846 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:31.027816    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:31.028649 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:31.028626    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 19:58:43.508516 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:43.508475    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:58:43.699252 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:43.699214    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:58:44.999233 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:44.999199    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:58:45.248608 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:45.248571    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:58:48.060672 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:48.060635    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:58:48.737883 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:58:48.737841    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:58:57.357404 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:58:57.357363    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:59:13.166755 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:13.166723    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:13.408373 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:13.408337    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:13.667276 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:13.667233    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:13.864618 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:13.864577    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:14.043419 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:14.043389    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:14.246567 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:14.246533    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:14.416200 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:14.416112    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:14.821759 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:14.821726    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:15.003199 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:15.003162    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:15.250215 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:15.250121    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:16.411216 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:16.411176    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:16.651308 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:16.651261    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:17.602569 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:17.602534    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:17.906476 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:17.906391    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:18.241335 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:18.241234    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:18.425891 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:18.425862    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:18.613653 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:18.613612    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:19.252807 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:19.252763    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:19.434359 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:19.434321    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:19.708087 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:19.708048    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:19.884648 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:19.884608    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:20.063717 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:20.063673    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:20.593789 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:20.593745    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:20.783108 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:20.783078    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:21.598415 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:21.598381    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:21.819476 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:21.819438    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:23.786478 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:23.786440    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:23.985727 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:23.985693    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:24.168994 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:24.168907    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:24.343257 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:24.343224    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:24.533546 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:24.533510    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:26.158805 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:26.158756    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:26.346528 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:26.346497    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:27.358098 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:59:27.358066    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:59:28.503716 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:28.503671    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:28.678759 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:28.678719    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:28.860484 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:28.860448    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:29.052034 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:29.052005    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:29.254863 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:29.254747    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:29.527388 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:29.527354    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:29.707055 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:29.707027    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:29.886840 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:29.886740    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:30.937272 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:30.937237    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:31.120217 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:31.120177    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:32.761492 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:32.761458    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 19:59:34.895369 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:34.895334    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:35.068183 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:35.068124    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:35.382463 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:35.382422    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:39.847816 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:39.847761    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:40.019193 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:40.019158    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:40.198476 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:40.198393    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:41.445197 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:41.445151    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 19:59:45.233922 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:45.233879    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:45.415307 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:45.415269    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:45.592543 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:45.592508    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:48.387082 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:48.387025    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:49.419013 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:49.418970    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:50.210933 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:50.210899    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:50.428127 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:50.428082    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:50.602049 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:50.602006    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:55.448691 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:55.448654    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 19:59:57.356761 ip-10-0-134-196 kubenswrapper[5224]: E1102 19:59:57.356724    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 19:59:59.231500 ip-10-0-134-196 kubenswrapper[5224]: I1102 19:59:59.231470    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:00:00.166421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.166381    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"]
Nov 02 20:00:00.169826 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.169803    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"
Nov 02 20:00:00.169826 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.169810    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"]
Nov 02 20:00:00.173175 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.173158    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"]
Nov 02 20:00:00.173288 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.173271    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"
Nov 02 20:00:00.176372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.176355    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"
Nov 02 20:00:00.223755 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.223726    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:00:00.223885 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.223732    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:00:00.223885 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.223811    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:00:00.260744 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.260716    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"]
Nov 02 20:00:00.276382 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.276350    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"]
Nov 02 20:00:00.277628 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.277604    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"]
Nov 02 20:00:00.297309 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.297278    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txb55\" (UniqueName: \"kubernetes.io/projected/c8e71803-122d-4040-8e09-f61586343778-kube-api-access-txb55\") pod \"maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7\" (UID: \"c8e71803-122d-4040-8e09-f61586343778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"
Nov 02 20:00:00.297425 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.297316    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc9r9\" (UniqueName: \"kubernetes.io/projected/845c0067-583e-47df-a1cb-18bdaf339277-kube-api-access-wc9r9\") pod \"maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6\" (UID: \"845c0067-583e-47df-a1cb-18bdaf339277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"
Nov 02 20:00:00.297494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.297428    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e71803-122d-4040-8e09-f61586343778-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7\" (UID: \"c8e71803-122d-4040-8e09-f61586343778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"
Nov 02 20:00:00.297494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.297455    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bxs2\" (UniqueName: \"kubernetes.io/projected/d61b95a5-0ca3-4180-be81-63dd44f253d6-kube-api-access-2bxs2\") pod \"maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22\" (UID: \"d61b95a5-0ca3-4180-be81-63dd44f253d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"
Nov 02 20:00:00.297494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.297482    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d61b95a5-0ca3-4180-be81-63dd44f253d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22\" (UID: \"d61b95a5-0ca3-4180-be81-63dd44f253d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"
Nov 02 20:00:00.297651 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.297529    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845c0067-583e-47df-a1cb-18bdaf339277-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6\" (UID: \"845c0067-583e-47df-a1cb-18bdaf339277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"
Nov 02 20:00:00.397990 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.397956    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc9r9\" (UniqueName: \"kubernetes.io/projected/845c0067-583e-47df-a1cb-18bdaf339277-kube-api-access-wc9r9\") pod \"maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6\" (UID: \"845c0067-583e-47df-a1cb-18bdaf339277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"
Nov 02 20:00:00.398198 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.398022    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e71803-122d-4040-8e09-f61586343778-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7\" (UID: \"c8e71803-122d-4040-8e09-f61586343778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"
Nov 02 20:00:00.398198 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.398040    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxs2\" (UniqueName: \"kubernetes.io/projected/d61b95a5-0ca3-4180-be81-63dd44f253d6-kube-api-access-2bxs2\") pod \"maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22\" (UID: \"d61b95a5-0ca3-4180-be81-63dd44f253d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"
Nov 02 20:00:00.398198 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.398063    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d61b95a5-0ca3-4180-be81-63dd44f253d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22\" (UID: \"d61b95a5-0ca3-4180-be81-63dd44f253d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"
Nov 02 20:00:00.398198 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.398079    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845c0067-583e-47df-a1cb-18bdaf339277-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6\" (UID: \"845c0067-583e-47df-a1cb-18bdaf339277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"
Nov 02 20:00:00.398198 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.398127    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txb55\" (UniqueName: \"kubernetes.io/projected/c8e71803-122d-4040-8e09-f61586343778-kube-api-access-txb55\") pod \"maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7\" (UID: \"c8e71803-122d-4040-8e09-f61586343778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"
Nov 02 20:00:00.398805 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.398782    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d61b95a5-0ca3-4180-be81-63dd44f253d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22\" (UID: \"d61b95a5-0ca3-4180-be81-63dd44f253d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"
Nov 02 20:00:00.398855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.398788    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845c0067-583e-47df-a1cb-18bdaf339277-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6\" (UID: \"845c0067-583e-47df-a1cb-18bdaf339277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"
Nov 02 20:00:00.398889 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.398865    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e71803-122d-4040-8e09-f61586343778-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7\" (UID: \"c8e71803-122d-4040-8e09-f61586343778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"
Nov 02 20:00:00.456824 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.456789    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bxs2\" (UniqueName: \"kubernetes.io/projected/d61b95a5-0ca3-4180-be81-63dd44f253d6-kube-api-access-2bxs2\") pod \"maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22\" (UID: \"d61b95a5-0ca3-4180-be81-63dd44f253d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"
Nov 02 20:00:00.456986 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.456970    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc9r9\" (UniqueName: \"kubernetes.io/projected/845c0067-583e-47df-a1cb-18bdaf339277-kube-api-access-wc9r9\") pod \"maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6\" (UID: \"845c0067-583e-47df-a1cb-18bdaf339277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"
Nov 02 20:00:00.457132 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.457112    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txb55\" (UniqueName: \"kubernetes.io/projected/c8e71803-122d-4040-8e09-f61586343778-kube-api-access-txb55\") pod \"maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7\" (UID: \"c8e71803-122d-4040-8e09-f61586343778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"
Nov 02 20:00:00.483842 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.483817    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"
Nov 02 20:00:00.489795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.489763    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"
Nov 02 20:00:00.495431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.495409    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"
Nov 02 20:00:00.566736 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.566412    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:00.871074 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:00:00.871043    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845c0067_583e_47df_a1cb_18bdaf339277.slice/crio-46629e8f3d2b906e131a1f6d79aa170adbc9e872c102aec7070dfd86726a16b3 WatchSource:0}: Error finding container 46629e8f3d2b906e131a1f6d79aa170adbc9e872c102aec7070dfd86726a16b3: Status 404 returned error can't find the container with id 46629e8f3d2b906e131a1f6d79aa170adbc9e872c102aec7070dfd86726a16b3
Nov 02 20:00:00.872865 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.872848    5224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:00:00.873828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.873805    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"]
Nov 02 20:00:00.917633 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:00:00.917602    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61b95a5_0ca3_4180_be81_63dd44f253d6.slice/crio-6d6d4a8e0be171d67cc0789c724b52334c932585993efffe568cc3d2acfd7968 WatchSource:0}: Error finding container 6d6d4a8e0be171d67cc0789c724b52334c932585993efffe568cc3d2acfd7968: Status 404 returned error can't find the container with id 6d6d4a8e0be171d67cc0789c724b52334c932585993efffe568cc3d2acfd7968
Nov 02 20:00:00.920120 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.920097    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"]
Nov 02 20:00:00.920466 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:00:00.920444    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e71803_122d_4040_8e09_f61586343778.slice/crio-a3ac73e09635b58d57fdc5ee689c4ff0fa1a504dc20153a2f2e7accca01200d1 WatchSource:0}: Error finding container a3ac73e09635b58d57fdc5ee689c4ff0fa1a504dc20153a2f2e7accca01200d1: Status 404 returned error can't find the container with id a3ac73e09635b58d57fdc5ee689c4ff0fa1a504dc20153a2f2e7accca01200d1
Nov 02 20:00:00.922872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:00.922851    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"]
Nov 02 20:00:01.337981 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:01.337944    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6" event={"ID":"845c0067-583e-47df-a1cb-18bdaf339277","Type":"ContainerStarted","Data":"46629e8f3d2b906e131a1f6d79aa170adbc9e872c102aec7070dfd86726a16b3"}
Nov 02 20:00:01.339164 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:01.339133    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22" event={"ID":"d61b95a5-0ca3-4180-be81-63dd44f253d6","Type":"ContainerStarted","Data":"6d6d4a8e0be171d67cc0789c724b52334c932585993efffe568cc3d2acfd7968"}
Nov 02 20:00:01.340215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:01.340194    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7" event={"ID":"c8e71803-122d-4040-8e09-f61586343778","Type":"ContainerStarted","Data":"a3ac73e09635b58d57fdc5ee689c4ff0fa1a504dc20153a2f2e7accca01200d1"}
Nov 02 20:00:04.600493 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.600452    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"]
Nov 02 20:00:04.604353 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.604330    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"
Nov 02 20:00:04.655655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.655589    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"]
Nov 02 20:00:04.705162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.705129    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:00:04.740226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.740181    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmb2p\" (UniqueName: \"kubernetes.io/projected/9171ef77-e898-4895-9c8e-002d7b69a704-kube-api-access-nmb2p\") pod \"maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv\" (UID: \"9171ef77-e898-4895-9c8e-002d7b69a704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"
Nov 02 20:00:04.740418 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.740248    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv\" (UID: \"9171ef77-e898-4895-9c8e-002d7b69a704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"
Nov 02 20:00:04.740418 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.740285    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv\" (UID: \"9171ef77-e898-4895-9c8e-002d7b69a704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"
Nov 02 20:00:04.841614 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.841575    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv\" (UID: \"9171ef77-e898-4895-9c8e-002d7b69a704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"
Nov 02 20:00:04.841819 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.841669    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmb2p\" (UniqueName: \"kubernetes.io/projected/9171ef77-e898-4895-9c8e-002d7b69a704-kube-api-access-nmb2p\") pod \"maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv\" (UID: \"9171ef77-e898-4895-9c8e-002d7b69a704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"
Nov 02 20:00:04.841819 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.841726    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv\" (UID: \"9171ef77-e898-4895-9c8e-002d7b69a704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"
Nov 02 20:00:04.842344 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.842317    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv\" (UID: \"9171ef77-e898-4895-9c8e-002d7b69a704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"
Nov 02 20:00:04.846332 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.846310    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv\" (UID: \"9171ef77-e898-4895-9c8e-002d7b69a704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"
Nov 02 20:00:04.900229 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.900154    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmb2p\" (UniqueName: \"kubernetes.io/projected/9171ef77-e898-4895-9c8e-002d7b69a704-kube-api-access-nmb2p\") pod \"maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv\" (UID: \"9171ef77-e898-4895-9c8e-002d7b69a704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"
Nov 02 20:00:04.921087 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:04.920890    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"
Nov 02 20:00:05.340358 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:00:05.337976    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9171ef77_e898_4895_9c8e_002d7b69a704.slice/crio-41001c93db738d474e8e38b940157ac4add77fef01583f3a3f2c36b5d46e008e WatchSource:0}: Error finding container 41001c93db738d474e8e38b940157ac4add77fef01583f3a3f2c36b5d46e008e: Status 404 returned error can't find the container with id 41001c93db738d474e8e38b940157ac4add77fef01583f3a3f2c36b5d46e008e
Nov 02 20:00:05.340358 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:05.340280    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"]
Nov 02 20:00:05.364713 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:05.364679    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv" event={"ID":"9171ef77-e898-4895-9c8e-002d7b69a704","Type":"ContainerStarted","Data":"41001c93db738d474e8e38b940157ac4add77fef01583f3a3f2c36b5d46e008e"}
Nov 02 20:00:06.000340 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:06.000311    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:06.371011 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:06.370926    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6" event={"ID":"845c0067-583e-47df-a1cb-18bdaf339277","Type":"ContainerStarted","Data":"adcd153bb80e1274fe22512f1ca8228fe77fa889fcb0360b56e63a453c665d2f"}
Nov 02 20:00:06.372526 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:06.372493    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22" event={"ID":"d61b95a5-0ca3-4180-be81-63dd44f253d6","Type":"ContainerStarted","Data":"6857b281006a34430b0551aa9fb18d2e3c4b82befab374efb7f12b4ef6d95d04"}
Nov 02 20:00:06.373858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:06.373837    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7" event={"ID":"c8e71803-122d-4040-8e09-f61586343778","Type":"ContainerStarted","Data":"62ba8bc95805ca02890d4a73963703b264b4fa4aa07d3706287796f4227cb8b3"}
Nov 02 20:00:06.485482 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:06.485431    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6" podStartSLOduration=1.588717017 podStartE2EDuration="6.4854199s" totalImagesPullingTime="4.896702883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:06.48519521 +0000 UTC m=+687.482883819" watchObservedRunningTime="2026-11-02 20:00:06.4854199 +0000 UTC m=+687.483108509"
Nov 02 20:00:06.602032 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:06.601983    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7" podStartSLOduration=1.7453100670000001 podStartE2EDuration="6.60196784s" totalImagesPullingTime="4.856657773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:06.601418976 +0000 UTC m=+687.599107584" watchObservedRunningTime="2026-11-02 20:00:06.60196784 +0000 UTC m=+687.599656449"
Nov 02 20:00:06.716188 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:06.716140    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22" podStartSLOduration=1.798225593 podStartE2EDuration="6.716125635s" totalImagesPullingTime="4.917900042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:06.715849877 +0000 UTC m=+687.713538489" watchObservedRunningTime="2026-11-02 20:00:06.716125635 +0000 UTC m=+687.713814244"
Nov 02 20:00:07.380336 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:07.380302    5224 generic.go:356] "Generic (PLEG): container finished" podID="845c0067-583e-47df-a1cb-18bdaf339277" containerID="adcd153bb80e1274fe22512f1ca8228fe77fa889fcb0360b56e63a453c665d2f" exitCode=60
Nov 02 20:00:07.380746 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:07.380381    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6" event={"ID":"845c0067-583e-47df-a1cb-18bdaf339277","Type":"ContainerDied","Data":"adcd153bb80e1274fe22512f1ca8228fe77fa889fcb0360b56e63a453c665d2f"}
Nov 02 20:00:07.380746 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:07.380582    5224 scope.go:122] "RemoveContainer" containerID="adcd153bb80e1274fe22512f1ca8228fe77fa889fcb0360b56e63a453c665d2f"
Nov 02 20:00:07.382091 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:07.382069    5224 generic.go:356] "Generic (PLEG): container finished" podID="d61b95a5-0ca3-4180-be81-63dd44f253d6" containerID="6857b281006a34430b0551aa9fb18d2e3c4b82befab374efb7f12b4ef6d95d04" exitCode=60
Nov 02 20:00:07.382207 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:07.382149    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22" event={"ID":"d61b95a5-0ca3-4180-be81-63dd44f253d6","Type":"ContainerDied","Data":"6857b281006a34430b0551aa9fb18d2e3c4b82befab374efb7f12b4ef6d95d04"}
Nov 02 20:00:07.382411 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:07.382396    5224 scope.go:122] "RemoveContainer" containerID="6857b281006a34430b0551aa9fb18d2e3c4b82befab374efb7f12b4ef6d95d04"
Nov 02 20:00:07.383865 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:07.383843    5224 generic.go:356] "Generic (PLEG): container finished" podID="c8e71803-122d-4040-8e09-f61586343778" containerID="62ba8bc95805ca02890d4a73963703b264b4fa4aa07d3706287796f4227cb8b3" exitCode=60
Nov 02 20:00:07.383949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:07.383875    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7" event={"ID":"c8e71803-122d-4040-8e09-f61586343778","Type":"ContainerDied","Data":"62ba8bc95805ca02890d4a73963703b264b4fa4aa07d3706287796f4227cb8b3"}
Nov 02 20:00:07.384089 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:07.384076    5224 scope.go:122] "RemoveContainer" containerID="62ba8bc95805ca02890d4a73963703b264b4fa4aa07d3706287796f4227cb8b3"
Nov 02 20:00:08.392404 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.392366    5224 generic.go:356] "Generic (PLEG): container finished" podID="c8e71803-122d-4040-8e09-f61586343778" containerID="c07aca1a2edbddb7549df246102f521e3507004531610fe6d0e769141adec8ee" exitCode=60
Nov 02 20:00:08.392894 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.392451    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7" event={"ID":"c8e71803-122d-4040-8e09-f61586343778","Type":"ContainerDied","Data":"c07aca1a2edbddb7549df246102f521e3507004531610fe6d0e769141adec8ee"}
Nov 02 20:00:08.392894 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.392512    5224 scope.go:122] "RemoveContainer" containerID="62ba8bc95805ca02890d4a73963703b264b4fa4aa07d3706287796f4227cb8b3"
Nov 02 20:00:08.392894 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.392797    5224 scope.go:122] "RemoveContainer" containerID="c07aca1a2edbddb7549df246102f521e3507004531610fe6d0e769141adec8ee"
Nov 02 20:00:08.393088 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:00:08.393057    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7_odh-ai-gateway-infra(c8e71803-122d-4040-8e09-f61586343778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7" podUID="c8e71803-122d-4040-8e09-f61586343778"
Nov 02 20:00:08.394288 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.394268    5224 generic.go:356] "Generic (PLEG): container finished" podID="9171ef77-e898-4895-9c8e-002d7b69a704" containerID="61de71598422feaa6b40a43adbdafb470ab1454b8fe2b6f3b8c2f59647d889a6" exitCode=0
Nov 02 20:00:08.394386 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.394348    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv" event={"ID":"9171ef77-e898-4895-9c8e-002d7b69a704","Type":"ContainerDied","Data":"61de71598422feaa6b40a43adbdafb470ab1454b8fe2b6f3b8c2f59647d889a6"}
Nov 02 20:00:08.396216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.396186    5224 generic.go:356] "Generic (PLEG): container finished" podID="845c0067-583e-47df-a1cb-18bdaf339277" containerID="b8da69a3bc6a06db1cce76ca9120d58ddf8ec3ddb775ce90037af99f1f22ca28" exitCode=60
Nov 02 20:00:08.396313 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.396262    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6" event={"ID":"845c0067-583e-47df-a1cb-18bdaf339277","Type":"ContainerDied","Data":"b8da69a3bc6a06db1cce76ca9120d58ddf8ec3ddb775ce90037af99f1f22ca28"}
Nov 02 20:00:08.396501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.396482    5224 scope.go:122] "RemoveContainer" containerID="b8da69a3bc6a06db1cce76ca9120d58ddf8ec3ddb775ce90037af99f1f22ca28"
Nov 02 20:00:08.396731 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:00:08.396704    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6_odh-ai-gateway-infra(845c0067-583e-47df-a1cb-18bdaf339277)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6" podUID="845c0067-583e-47df-a1cb-18bdaf339277"
Nov 02 20:00:08.402197 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.402142    5224 generic.go:356] "Generic (PLEG): container finished" podID="d61b95a5-0ca3-4180-be81-63dd44f253d6" containerID="6aadf7712ca8fc8e2888f65c8706f2e5a16de6aa57eeae6b143d7dce056c43a6" exitCode=60
Nov 02 20:00:08.402287 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.402201    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22" event={"ID":"d61b95a5-0ca3-4180-be81-63dd44f253d6","Type":"ContainerDied","Data":"6aadf7712ca8fc8e2888f65c8706f2e5a16de6aa57eeae6b143d7dce056c43a6"}
Nov 02 20:00:08.402471 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.402457    5224 scope.go:122] "RemoveContainer" containerID="6aadf7712ca8fc8e2888f65c8706f2e5a16de6aa57eeae6b143d7dce056c43a6"
Nov 02 20:00:08.402736 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:00:08.402717    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22_odh-ai-gateway-infra(d61b95a5-0ca3-4180-be81-63dd44f253d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22" podUID="d61b95a5-0ca3-4180-be81-63dd44f253d6"
Nov 02 20:00:08.405672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.405645    5224 scope.go:122] "RemoveContainer" containerID="adcd153bb80e1274fe22512f1ca8228fe77fa889fcb0360b56e63a453c665d2f"
Nov 02 20:00:08.418077 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:08.418055    5224 scope.go:122] "RemoveContainer" containerID="6857b281006a34430b0551aa9fb18d2e3c4b82befab374efb7f12b4ef6d95d04"
Nov 02 20:00:09.409500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:09.409460    5224 scope.go:122] "RemoveContainer" containerID="c07aca1a2edbddb7549df246102f521e3507004531610fe6d0e769141adec8ee"
Nov 02 20:00:09.410457 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:00:09.410428    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7_odh-ai-gateway-infra(c8e71803-122d-4040-8e09-f61586343778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7" podUID="c8e71803-122d-4040-8e09-f61586343778"
Nov 02 20:00:09.416932 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:09.416907    5224 scope.go:122] "RemoveContainer" containerID="b8da69a3bc6a06db1cce76ca9120d58ddf8ec3ddb775ce90037af99f1f22ca28"
Nov 02 20:00:09.417126 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:00:09.417107    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6_odh-ai-gateway-infra(845c0067-583e-47df-a1cb-18bdaf339277)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6" podUID="845c0067-583e-47df-a1cb-18bdaf339277"
Nov 02 20:00:09.418641 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:09.418357    5224 scope.go:122] "RemoveContainer" containerID="6aadf7712ca8fc8e2888f65c8706f2e5a16de6aa57eeae6b143d7dce056c43a6"
Nov 02 20:00:09.418641 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:00:09.418497    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22_odh-ai-gateway-infra(d61b95a5-0ca3-4180-be81-63dd44f253d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22" podUID="d61b95a5-0ca3-4180-be81-63dd44f253d6"
Nov 02 20:00:09.550717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:09.550694    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"
Nov 02 20:00:09.703561 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:09.703478    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-xks-ca\") pod \"9171ef77-e898-4895-9c8e-002d7b69a704\" (UID: \"9171ef77-e898-4895-9c8e-002d7b69a704\") "
Nov 02 20:00:09.703561 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:09.703536    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9171ef77-e898-4895-9c8e-002d7b69a704-kube-api-access-nmb2p\" (UniqueName: \"kubernetes.io/projected/9171ef77-e898-4895-9c8e-002d7b69a704-kube-api-access-nmb2p\") pod \"9171ef77-e898-4895-9c8e-002d7b69a704\" (UID: \"9171ef77-e898-4895-9c8e-002d7b69a704\") "
Nov 02 20:00:09.703733 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:09.703567    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-openshift-ca\") pod \"9171ef77-e898-4895-9c8e-002d7b69a704\" (UID: \"9171ef77-e898-4895-9c8e-002d7b69a704\") "
Nov 02 20:00:09.704010 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:09.703983    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-openshift-ca" pod "9171ef77-e898-4895-9c8e-002d7b69a704" (UID: "9171ef77-e898-4895-9c8e-002d7b69a704"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:00:09.705704 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:09.705681    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-xks-ca" pod "9171ef77-e898-4895-9c8e-002d7b69a704" (UID: "9171ef77-e898-4895-9c8e-002d7b69a704"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:00:09.705778 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:09.705732    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9171ef77-e898-4895-9c8e-002d7b69a704-kube-api-access-nmb2p" pod "9171ef77-e898-4895-9c8e-002d7b69a704" (UID: "9171ef77-e898-4895-9c8e-002d7b69a704"). InnerVolumeSpecName "kube-api-access-nmb2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:00:09.804838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:09.804809    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:09.804838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:09.804835    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmb2p\" (UniqueName: \"kubernetes.io/projected/9171ef77-e898-4895-9c8e-002d7b69a704-kube-api-access-nmb2p\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:09.804838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:09.804847    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9171ef77-e898-4895-9c8e-002d7b69a704-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:10.424988 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:10.424955    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"
Nov 02 20:00:10.425484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:10.424996    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv" event={"ID":"9171ef77-e898-4895-9c8e-002d7b69a704","Type":"ContainerDied","Data":"41001c93db738d474e8e38b940157ac4add77fef01583f3a3f2c36b5d46e008e"}
Nov 02 20:00:10.425484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:10.425045    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41001c93db738d474e8e38b940157ac4add77fef01583f3a3f2c36b5d46e008e"
Nov 02 20:00:11.065861 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:11.065826    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:11.997230 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:11.997195    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"]
Nov 02 20:00:11.997607 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:11.997590    5224 cpu_manager.go:414] "containerMap: removing container" podUID="9171ef77-e898-4895-9c8e-002d7b69a704" containerName="revoke-keys"
Nov 02 20:00:11.997607 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:11.997600    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9171ef77-e898-4895-9c8e-002d7b69a704" containerName="revoke-keys"
Nov 02 20:00:11.997698 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:11.997687    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9171ef77-e898-4895-9c8e-002d7b69a704" containerName="revoke-keys"
Nov 02 20:00:12.000839 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.000824    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"
Nov 02 20:00:12.054075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.054038    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"]
Nov 02 20:00:12.103271 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.103240    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:00:12.125212 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.125180    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz\" (UID: \"5908975c-8d31-40d2-841b-d9b88a5f483d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"
Nov 02 20:00:12.125370 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.125219    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz\" (UID: \"5908975c-8d31-40d2-841b-d9b88a5f483d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"
Nov 02 20:00:12.125370 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.125323    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv84q\" (UniqueName: \"kubernetes.io/projected/5908975c-8d31-40d2-841b-d9b88a5f483d-kube-api-access-qv84q\") pod \"maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz\" (UID: \"5908975c-8d31-40d2-841b-d9b88a5f483d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"
Nov 02 20:00:12.225933 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.225901    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz\" (UID: \"5908975c-8d31-40d2-841b-d9b88a5f483d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"
Nov 02 20:00:12.226103 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.225980    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv84q\" (UniqueName: \"kubernetes.io/projected/5908975c-8d31-40d2-841b-d9b88a5f483d-kube-api-access-qv84q\") pod \"maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz\" (UID: \"5908975c-8d31-40d2-841b-d9b88a5f483d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"
Nov 02 20:00:12.226103 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.226015    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz\" (UID: \"5908975c-8d31-40d2-841b-d9b88a5f483d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"
Nov 02 20:00:12.226582 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.226565    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz\" (UID: \"5908975c-8d31-40d2-841b-d9b88a5f483d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"
Nov 02 20:00:12.228176 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.228154    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz\" (UID: \"5908975c-8d31-40d2-841b-d9b88a5f483d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"
Nov 02 20:00:12.284056 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.284026    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv84q\" (UniqueName: \"kubernetes.io/projected/5908975c-8d31-40d2-841b-d9b88a5f483d-kube-api-access-qv84q\") pod \"maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz\" (UID: \"5908975c-8d31-40d2-841b-d9b88a5f483d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"
Nov 02 20:00:12.313863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.313834    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"
Nov 02 20:00:12.679348 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:00:12.679310    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5908975c_8d31_40d2_841b_d9b88a5f483d.slice/crio-43c44ce8f04df3c582c5f03e2670cd63bca4b83373b03984f0f052d6df8f091a WatchSource:0}: Error finding container 43c44ce8f04df3c582c5f03e2670cd63bca4b83373b03984f0f052d6df8f091a: Status 404 returned error can't find the container with id 43c44ce8f04df3c582c5f03e2670cd63bca4b83373b03984f0f052d6df8f091a
Nov 02 20:00:12.682254 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:12.682229    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"]
Nov 02 20:00:13.441007 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:13.440965    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz" event={"ID":"5908975c-8d31-40d2-841b-d9b88a5f483d","Type":"ContainerStarted","Data":"3cfe4f2dd1f1aa60f34ea1c3516d5ed200b274e04c146890ba43873fbfa83bae"}
Nov 02 20:00:13.441396 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:13.441031    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz" event={"ID":"5908975c-8d31-40d2-841b-d9b88a5f483d","Type":"ContainerStarted","Data":"43c44ce8f04df3c582c5f03e2670cd63bca4b83373b03984f0f052d6df8f091a"}
Nov 02 20:00:13.557332 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:13.557277    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz" podStartSLOduration=2.5572622369999998 podStartE2EDuration="2.557262237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:13.555820151 +0000 UTC m=+694.553508760" watchObservedRunningTime="2026-11-02 20:00:13.557262237 +0000 UTC m=+694.554950843"
Nov 02 20:00:14.447902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:14.447863    5224 generic.go:356] "Generic (PLEG): container finished" podID="5908975c-8d31-40d2-841b-d9b88a5f483d" containerID="3cfe4f2dd1f1aa60f34ea1c3516d5ed200b274e04c146890ba43873fbfa83bae" exitCode=0
Nov 02 20:00:14.447902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:14.447904    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz" event={"ID":"5908975c-8d31-40d2-841b-d9b88a5f483d","Type":"ContainerDied","Data":"3cfe4f2dd1f1aa60f34ea1c3516d5ed200b274e04c146890ba43873fbfa83bae"}
Nov 02 20:00:15.578752 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:15.578723    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"
Nov 02 20:00:15.758969 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:15.758873    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-xks-ca\") pod \"5908975c-8d31-40d2-841b-d9b88a5f483d\" (UID: \"5908975c-8d31-40d2-841b-d9b88a5f483d\") "
Nov 02 20:00:15.758969 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:15.758970    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5908975c-8d31-40d2-841b-d9b88a5f483d-kube-api-access-qv84q\" (UniqueName: \"kubernetes.io/projected/5908975c-8d31-40d2-841b-d9b88a5f483d-kube-api-access-qv84q\") pod \"5908975c-8d31-40d2-841b-d9b88a5f483d\" (UID: \"5908975c-8d31-40d2-841b-d9b88a5f483d\") "
Nov 02 20:00:15.759227 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:15.759004    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-openshift-ca\") pod \"5908975c-8d31-40d2-841b-d9b88a5f483d\" (UID: \"5908975c-8d31-40d2-841b-d9b88a5f483d\") "
Nov 02 20:00:15.759431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:15.759404    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-openshift-ca" pod "5908975c-8d31-40d2-841b-d9b88a5f483d" (UID: "5908975c-8d31-40d2-841b-d9b88a5f483d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:00:15.761050 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:15.761023    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-xks-ca" pod "5908975c-8d31-40d2-841b-d9b88a5f483d" (UID: "5908975c-8d31-40d2-841b-d9b88a5f483d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:00:15.761154 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:15.761113    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5908975c-8d31-40d2-841b-d9b88a5f483d-kube-api-access-qv84q" pod "5908975c-8d31-40d2-841b-d9b88a5f483d" (UID: "5908975c-8d31-40d2-841b-d9b88a5f483d"). InnerVolumeSpecName "kube-api-access-qv84q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:00:15.860327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:15.860282    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:15.860327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:15.860317    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv84q\" (UniqueName: \"kubernetes.io/projected/5908975c-8d31-40d2-841b-d9b88a5f483d-kube-api-access-qv84q\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:15.860327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:15.860328    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5908975c-8d31-40d2-841b-d9b88a5f483d-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:16.331736 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:16.331700    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:16.458543 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:16.458515    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"
Nov 02 20:00:16.458716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:16.458546    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz" event={"ID":"5908975c-8d31-40d2-841b-d9b88a5f483d","Type":"ContainerDied","Data":"43c44ce8f04df3c582c5f03e2670cd63bca4b83373b03984f0f052d6df8f091a"}
Nov 02 20:00:16.458716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:16.458589    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43c44ce8f04df3c582c5f03e2670cd63bca4b83373b03984f0f052d6df8f091a"
Nov 02 20:00:16.532146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:16.532110    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:16.709612 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:16.709528    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:20.161424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:20.161393    5224 scope.go:122] "RemoveContainer" containerID="6aadf7712ca8fc8e2888f65c8706f2e5a16de6aa57eeae6b143d7dce056c43a6"
Nov 02 20:00:20.479421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:20.479387    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22" event={"ID":"d61b95a5-0ca3-4180-be81-63dd44f253d6","Type":"ContainerStarted","Data":"5ef11d27ef9c3332f2fc28da19bd5122603a6f853049d9f44879141cf5109650"}
Nov 02 20:00:21.161179 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:21.161145    5224 scope.go:122] "RemoveContainer" containerID="b8da69a3bc6a06db1cce76ca9120d58ddf8ec3ddb775ce90037af99f1f22ca28"
Nov 02 20:00:21.161381 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:21.161232    5224 scope.go:122] "RemoveContainer" containerID="c07aca1a2edbddb7549df246102f521e3507004531610fe6d0e769141adec8ee"
Nov 02 20:00:21.487133 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:21.487088    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6" event={"ID":"845c0067-583e-47df-a1cb-18bdaf339277","Type":"ContainerStarted","Data":"7ae18a60cd4ebe84ae8e2b9bff2bb2a245e79220746667d272d4f5b111e73f69"}
Nov 02 20:00:21.488956 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:21.488925    5224 generic.go:356] "Generic (PLEG): container finished" podID="d61b95a5-0ca3-4180-be81-63dd44f253d6" containerID="5ef11d27ef9c3332f2fc28da19bd5122603a6f853049d9f44879141cf5109650" exitCode=60
Nov 02 20:00:21.489080 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:21.488992    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22" event={"ID":"d61b95a5-0ca3-4180-be81-63dd44f253d6","Type":"ContainerDied","Data":"5ef11d27ef9c3332f2fc28da19bd5122603a6f853049d9f44879141cf5109650"}
Nov 02 20:00:21.489080 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:21.489044    5224 scope.go:122] "RemoveContainer" containerID="6aadf7712ca8fc8e2888f65c8706f2e5a16de6aa57eeae6b143d7dce056c43a6"
Nov 02 20:00:21.489346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:21.489329    5224 scope.go:122] "RemoveContainer" containerID="5ef11d27ef9c3332f2fc28da19bd5122603a6f853049d9f44879141cf5109650"
Nov 02 20:00:21.489574 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:00:21.489555    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22_odh-ai-gateway-infra(d61b95a5-0ca3-4180-be81-63dd44f253d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22" podUID="d61b95a5-0ca3-4180-be81-63dd44f253d6"
Nov 02 20:00:21.490937 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:21.490907    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7" event={"ID":"c8e71803-122d-4040-8e09-f61586343778","Type":"ContainerStarted","Data":"9f1ec38614a52058f6bc0ae18f0ea2369547d9ef7a347b210a7908457461d0b2"}
Nov 02 20:00:21.603399 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:21.603361    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"]
Nov 02 20:00:21.781071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:21.781028    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:22.497782 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.497730    5224 generic.go:356] "Generic (PLEG): container finished" podID="845c0067-583e-47df-a1cb-18bdaf339277" containerID="7ae18a60cd4ebe84ae8e2b9bff2bb2a245e79220746667d272d4f5b111e73f69" exitCode=60
Nov 02 20:00:22.498247 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.497807    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6" event={"ID":"845c0067-583e-47df-a1cb-18bdaf339277","Type":"ContainerDied","Data":"7ae18a60cd4ebe84ae8e2b9bff2bb2a245e79220746667d272d4f5b111e73f69"}
Nov 02 20:00:22.498247 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.497867    5224 scope.go:122] "RemoveContainer" containerID="b8da69a3bc6a06db1cce76ca9120d58ddf8ec3ddb775ce90037af99f1f22ca28"
Nov 02 20:00:22.498247 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.498168    5224 scope.go:122] "RemoveContainer" containerID="7ae18a60cd4ebe84ae8e2b9bff2bb2a245e79220746667d272d4f5b111e73f69"
Nov 02 20:00:22.498437 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:00:22.498414    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6_odh-ai-gateway-infra(845c0067-583e-47df-a1cb-18bdaf339277)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6" podUID="845c0067-583e-47df-a1cb-18bdaf339277"
Nov 02 20:00:22.506028 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.506002    5224 generic.go:356] "Generic (PLEG): container finished" podID="c8e71803-122d-4040-8e09-f61586343778" containerID="9f1ec38614a52058f6bc0ae18f0ea2369547d9ef7a347b210a7908457461d0b2" exitCode=60
Nov 02 20:00:22.506159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.506077    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7" event={"ID":"c8e71803-122d-4040-8e09-f61586343778","Type":"ContainerDied","Data":"9f1ec38614a52058f6bc0ae18f0ea2369547d9ef7a347b210a7908457461d0b2"}
Nov 02 20:00:22.506422 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.506405    5224 scope.go:122] "RemoveContainer" containerID="9f1ec38614a52058f6bc0ae18f0ea2369547d9ef7a347b210a7908457461d0b2"
Nov 02 20:00:22.511802 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:00:22.507166    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7_odh-ai-gateway-infra(c8e71803-122d-4040-8e09-f61586343778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7" podUID="c8e71803-122d-4040-8e09-f61586343778"
Nov 02 20:00:22.514373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.513968    5224 scope.go:122] "RemoveContainer" containerID="c07aca1a2edbddb7549df246102f521e3507004531610fe6d0e769141adec8ee"
Nov 02 20:00:22.613973 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.613937    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"]
Nov 02 20:00:22.646963 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.646935    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"
Nov 02 20:00:22.721077 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.721042    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61b95a5-0ca3-4180-be81-63dd44f253d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d61b95a5-0ca3-4180-be81-63dd44f253d6-maas-api-service-ca\") pod \"d61b95a5-0ca3-4180-be81-63dd44f253d6\" (UID: \"d61b95a5-0ca3-4180-be81-63dd44f253d6\") "
Nov 02 20:00:22.721077 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.721090    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61b95a5-0ca3-4180-be81-63dd44f253d6-kube-api-access-2bxs2\" (UniqueName: \"kubernetes.io/projected/d61b95a5-0ca3-4180-be81-63dd44f253d6-kube-api-access-2bxs2\") pod \"d61b95a5-0ca3-4180-be81-63dd44f253d6\" (UID: \"d61b95a5-0ca3-4180-be81-63dd44f253d6\") "
Nov 02 20:00:22.721445 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.721420    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61b95a5-0ca3-4180-be81-63dd44f253d6-maas-api-service-ca" pod "d61b95a5-0ca3-4180-be81-63dd44f253d6" (UID: "d61b95a5-0ca3-4180-be81-63dd44f253d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:00:22.723325 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.723305    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61b95a5-0ca3-4180-be81-63dd44f253d6-kube-api-access-2bxs2" pod "d61b95a5-0ca3-4180-be81-63dd44f253d6" (UID: "d61b95a5-0ca3-4180-be81-63dd44f253d6"). InnerVolumeSpecName "kube-api-access-2bxs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:00:22.821841 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.821805    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d61b95a5-0ca3-4180-be81-63dd44f253d6-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:22.821841 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.821835    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bxs2\" (UniqueName: \"kubernetes.io/projected/d61b95a5-0ca3-4180-be81-63dd44f253d6-kube-api-access-2bxs2\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:22.842137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:22.842106    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"]
Nov 02 20:00:23.490137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.490053    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:23.520091 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.520068    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"
Nov 02 20:00:23.520501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.520067    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22" event={"ID":"d61b95a5-0ca3-4180-be81-63dd44f253d6","Type":"ContainerDied","Data":"6d6d4a8e0be171d67cc0789c724b52334c932585993efffe568cc3d2acfd7968"}
Nov 02 20:00:23.520501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.520169    5224 scope.go:122] "RemoveContainer" containerID="5ef11d27ef9c3332f2fc28da19bd5122603a6f853049d9f44879141cf5109650"
Nov 02 20:00:23.659470 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.659436    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:23.680976 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.680952    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"
Nov 02 20:00:23.683204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.683180    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"
Nov 02 20:00:23.730690 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.730649    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e71803-122d-4040-8e09-f61586343778-kube-api-access-txb55\" (UniqueName: \"kubernetes.io/projected/c8e71803-122d-4040-8e09-f61586343778-kube-api-access-txb55\") pod \"c8e71803-122d-4040-8e09-f61586343778\" (UID: \"c8e71803-122d-4040-8e09-f61586343778\") "
Nov 02 20:00:23.730690 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.730698    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845c0067-583e-47df-a1cb-18bdaf339277-kube-api-access-wc9r9\" (UniqueName: \"kubernetes.io/projected/845c0067-583e-47df-a1cb-18bdaf339277-kube-api-access-wc9r9\") pod \"845c0067-583e-47df-a1cb-18bdaf339277\" (UID: \"845c0067-583e-47df-a1cb-18bdaf339277\") "
Nov 02 20:00:23.730976 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.730725    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845c0067-583e-47df-a1cb-18bdaf339277-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845c0067-583e-47df-a1cb-18bdaf339277-maas-api-service-ca\") pod \"845c0067-583e-47df-a1cb-18bdaf339277\" (UID: \"845c0067-583e-47df-a1cb-18bdaf339277\") "
Nov 02 20:00:23.730976 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.730752    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e71803-122d-4040-8e09-f61586343778-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e71803-122d-4040-8e09-f61586343778-maas-api-service-ca\") pod \"c8e71803-122d-4040-8e09-f61586343778\" (UID: \"c8e71803-122d-4040-8e09-f61586343778\") "
Nov 02 20:00:23.731182 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.731157    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e71803-122d-4040-8e09-f61586343778-maas-api-service-ca" pod "c8e71803-122d-4040-8e09-f61586343778" (UID: "c8e71803-122d-4040-8e09-f61586343778"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:00:23.731235 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.731155    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845c0067-583e-47df-a1cb-18bdaf339277-maas-api-service-ca" pod "845c0067-583e-47df-a1cb-18bdaf339277" (UID: "845c0067-583e-47df-a1cb-18bdaf339277"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:00:23.733091 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.733061    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845c0067-583e-47df-a1cb-18bdaf339277-kube-api-access-wc9r9" pod "845c0067-583e-47df-a1cb-18bdaf339277" (UID: "845c0067-583e-47df-a1cb-18bdaf339277"). InnerVolumeSpecName "kube-api-access-wc9r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:00:23.733210 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.733061    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e71803-122d-4040-8e09-f61586343778-kube-api-access-txb55" pod "c8e71803-122d-4040-8e09-f61586343778" (UID: "c8e71803-122d-4040-8e09-f61586343778"). InnerVolumeSpecName "kube-api-access-txb55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:00:23.751316 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.751287    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"]
Nov 02 20:00:23.754439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.754417    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-70ebea5b-29894160-tsw22"]
Nov 02 20:00:23.832054 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.832022    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txb55\" (UniqueName: \"kubernetes.io/projected/c8e71803-122d-4040-8e09-f61586343778-kube-api-access-txb55\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:23.832054 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.832050    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc9r9\" (UniqueName: \"kubernetes.io/projected/845c0067-583e-47df-a1cb-18bdaf339277-kube-api-access-wc9r9\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:23.832054 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.832061    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845c0067-583e-47df-a1cb-18bdaf339277-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:23.832288 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:23.832072    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e71803-122d-4040-8e09-f61586343778-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:24.529456 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.529431    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"
Nov 02 20:00:24.529938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.529430    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6" event={"ID":"845c0067-583e-47df-a1cb-18bdaf339277","Type":"ContainerDied","Data":"46629e8f3d2b906e131a1f6d79aa170adbc9e872c102aec7070dfd86726a16b3"}
Nov 02 20:00:24.529938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.529569    5224 scope.go:122] "RemoveContainer" containerID="7ae18a60cd4ebe84ae8e2b9bff2bb2a245e79220746667d272d4f5b111e73f69"
Nov 02 20:00:24.531884 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.531859    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7" event={"ID":"c8e71803-122d-4040-8e09-f61586343778","Type":"ContainerDied","Data":"a3ac73e09635b58d57fdc5ee689c4ff0fa1a504dc20153a2f2e7accca01200d1"}
Nov 02 20:00:24.531995 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.531900    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"
Nov 02 20:00:24.539526 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.539495    5224 scope.go:122] "RemoveContainer" containerID="9f1ec38614a52058f6bc0ae18f0ea2369547d9ef7a347b210a7908457461d0b2"
Nov 02 20:00:24.698032 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.698001    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"]
Nov 02 20:00:24.700366 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.700339    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4db200eb-29894160-r84m6"]
Nov 02 20:00:24.778760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.778728    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"]
Nov 02 20:00:24.779184 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779165    5224 cpu_manager.go:414] "containerMap: removing container" podUID="845c0067-583e-47df-a1cb-18bdaf339277" containerName="cleanup"
Nov 02 20:00:24.779274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779185    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845c0067-583e-47df-a1cb-18bdaf339277" containerName="cleanup"
Nov 02 20:00:24.779274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779203    5224 cpu_manager.go:414] "containerMap: removing container" podUID="c8e71803-122d-4040-8e09-f61586343778" containerName="cleanup"
Nov 02 20:00:24.779274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779210    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e71803-122d-4040-8e09-f61586343778" containerName="cleanup"
Nov 02 20:00:24.779274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779226    5224 cpu_manager.go:414] "containerMap: removing container" podUID="5908975c-8d31-40d2-841b-d9b88a5f483d" containerName="revoke-keys"
Nov 02 20:00:24.779274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779233    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5908975c-8d31-40d2-841b-d9b88a5f483d" containerName="revoke-keys"
Nov 02 20:00:24.779274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779244    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d61b95a5-0ca3-4180-be81-63dd44f253d6" containerName="cleanup"
Nov 02 20:00:24.779274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779252    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61b95a5-0ca3-4180-be81-63dd44f253d6" containerName="cleanup"
Nov 02 20:00:24.779274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779265    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d61b95a5-0ca3-4180-be81-63dd44f253d6" containerName="cleanup"
Nov 02 20:00:24.779274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779273    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61b95a5-0ca3-4180-be81-63dd44f253d6" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779286    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d61b95a5-0ca3-4180-be81-63dd44f253d6" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779293    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61b95a5-0ca3-4180-be81-63dd44f253d6" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779342    5224 cpu_manager.go:414] "containerMap: removing container" podUID="c8e71803-122d-4040-8e09-f61586343778" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779350    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e71803-122d-4040-8e09-f61586343778" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779359    5224 cpu_manager.go:414] "containerMap: removing container" podUID="845c0067-583e-47df-a1cb-18bdaf339277" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779367    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845c0067-583e-47df-a1cb-18bdaf339277" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779470    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="845c0067-583e-47df-a1cb-18bdaf339277" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779483    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e71803-122d-4040-8e09-f61586343778" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779491    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="845c0067-583e-47df-a1cb-18bdaf339277" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779499    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="845c0067-583e-47df-a1cb-18bdaf339277" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779508    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e71803-122d-4040-8e09-f61586343778" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779518    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5908975c-8d31-40d2-841b-d9b88a5f483d" containerName="revoke-keys"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779526    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61b95a5-0ca3-4180-be81-63dd44f253d6" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779535    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61b95a5-0ca3-4180-be81-63dd44f253d6" containerName="cleanup"
Nov 02 20:00:24.779724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.779543    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61b95a5-0ca3-4180-be81-63dd44f253d6" containerName="cleanup"
Nov 02 20:00:24.782914 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.782891    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"
Nov 02 20:00:24.812960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.812923    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"]
Nov 02 20:00:24.814820 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.814797    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-3acf2229-29894160-kqxs7"]
Nov 02 20:00:24.835577 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.835540    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"]
Nov 02 20:00:24.836441 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.836412    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:00:24.836566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.836437    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:00:24.836566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.836495    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:00:24.836566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.836546    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:00:24.839985 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.839960    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx\" (UID: \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"
Nov 02 20:00:24.840078 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.839994    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24glb\" (UniqueName: \"kubernetes.io/projected/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-kube-api-access-24glb\") pod \"maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx\" (UID: \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"
Nov 02 20:00:24.840123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.840109    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx\" (UID: \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"
Nov 02 20:00:24.941019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.940981    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx\" (UID: \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"
Nov 02 20:00:24.941019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.941023    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24glb\" (UniqueName: \"kubernetes.io/projected/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-kube-api-access-24glb\") pod \"maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx\" (UID: \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"
Nov 02 20:00:24.941212 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.941071    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx\" (UID: \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"
Nov 02 20:00:24.941677 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.941657    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx\" (UID: \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"
Nov 02 20:00:24.943506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.943483    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx\" (UID: \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"
Nov 02 20:00:24.999438 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:24.999404    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24glb\" (UniqueName: \"kubernetes.io/projected/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-kube-api-access-24glb\") pod \"maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx\" (UID: \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"
Nov 02 20:00:25.095879 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:25.095796    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"
Nov 02 20:00:25.167107 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:25.167070    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845c0067-583e-47df-a1cb-18bdaf339277" path="/var/lib/kubelet/pods/845c0067-583e-47df-a1cb-18bdaf339277/volumes"
Nov 02 20:00:25.167464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:25.167451    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e71803-122d-4040-8e09-f61586343778" path="/var/lib/kubelet/pods/c8e71803-122d-4040-8e09-f61586343778/volumes"
Nov 02 20:00:25.167805 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:25.167793    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61b95a5-0ca3-4180-be81-63dd44f253d6" path="/var/lib/kubelet/pods/d61b95a5-0ca3-4180-be81-63dd44f253d6/volumes"
Nov 02 20:00:25.461949 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:00:25.461908    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b8968a_cee0_4cce_9ef8_b63ecaf9b815.slice/crio-0e0271a81c39064d04caf5e1b614e786be164eb97aae7ab94aba0accb4f0fb07 WatchSource:0}: Error finding container 0e0271a81c39064d04caf5e1b614e786be164eb97aae7ab94aba0accb4f0fb07: Status 404 returned error can't find the container with id 0e0271a81c39064d04caf5e1b614e786be164eb97aae7ab94aba0accb4f0fb07
Nov 02 20:00:25.465128 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:25.465105    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"]
Nov 02 20:00:25.537605 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:25.537570    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx" event={"ID":"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815","Type":"ContainerStarted","Data":"0e0271a81c39064d04caf5e1b614e786be164eb97aae7ab94aba0accb4f0fb07"}
Nov 02 20:00:25.804681 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:25.804650    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:26.054782 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:26.054740    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:26.425229 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:26.425184    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:26.545431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:26.545396    5224 generic.go:356] "Generic (PLEG): container finished" podID="d6b8968a-cee0-4cce-9ef8-b63ecaf9b815" containerID="0a3d81ea410fb1c0131db9cdcac74bbdaf18ad09778948c01a708be9d02b4199" exitCode=0
Nov 02 20:00:26.545853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:26.545482    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx" event={"ID":"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815","Type":"ContainerDied","Data":"0a3d81ea410fb1c0131db9cdcac74bbdaf18ad09778948c01a708be9d02b4199"}
Nov 02 20:00:27.366661 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:00:27.366616    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:00:27.701439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:27.701407    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"
Nov 02 20:00:27.764417 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:27.764379    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-xks-ca\") pod \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\" (UID: \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\") "
Nov 02 20:00:27.764601 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:27.764428    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-kube-api-access-24glb\" (UniqueName: \"kubernetes.io/projected/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-kube-api-access-24glb\") pod \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\" (UID: \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\") "
Nov 02 20:00:27.764601 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:27.764508    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-openshift-ca\") pod \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\" (UID: \"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815\") "
Nov 02 20:00:27.764948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:27.764913    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-openshift-ca" pod "d6b8968a-cee0-4cce-9ef8-b63ecaf9b815" (UID: "d6b8968a-cee0-4cce-9ef8-b63ecaf9b815"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:00:27.766603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:27.766581    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-xks-ca" pod "d6b8968a-cee0-4cce-9ef8-b63ecaf9b815" (UID: "d6b8968a-cee0-4cce-9ef8-b63ecaf9b815"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:00:27.766711 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:27.766696    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-kube-api-access-24glb" pod "d6b8968a-cee0-4cce-9ef8-b63ecaf9b815" (UID: "d6b8968a-cee0-4cce-9ef8-b63ecaf9b815"). InnerVolumeSpecName "kube-api-access-24glb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:00:27.816203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:27.816163    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:27.865760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:27.865719    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:27.865760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:27.865754    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:27.865983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:27.865790    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24glb\" (UniqueName: \"kubernetes.io/projected/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815-kube-api-access-24glb\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:28.146395 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:28.146359    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:28.432458 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:28.432348    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:28.556191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:28.556159    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"
Nov 02 20:00:28.556368 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:28.556195    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx" event={"ID":"d6b8968a-cee0-4cce-9ef8-b63ecaf9b815","Type":"ContainerDied","Data":"0e0271a81c39064d04caf5e1b614e786be164eb97aae7ab94aba0accb4f0fb07"}
Nov 02 20:00:28.556368 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:28.556244    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e0271a81c39064d04caf5e1b614e786be164eb97aae7ab94aba0accb4f0fb07"
Nov 02 20:00:28.671316 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:28.671282    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"]
Nov 02 20:00:28.673620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:28.673590    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4db200eb-mkfrv"]
Nov 02 20:00:28.777714 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:28.777678    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:29.024921 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:29.024876    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:29.167506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:29.167429    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9171ef77-e898-4895-9c8e-002d7b69a704" path="/var/lib/kubelet/pods/9171ef77-e898-4895-9c8e-002d7b69a704/volumes"
Nov 02 20:00:29.461487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:29.461383    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:29.645867 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:29.645839    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:29.828031 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:29.827986    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:29.905057 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:29.905023    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"]
Nov 02 20:00:29.956567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:29.956528    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70ebea5b-mpmwz"]
Nov 02 20:00:30.142620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.142533    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:30.323493 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.323453    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:30.375908 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.375874    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"]
Nov 02 20:00:30.376266 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.376254    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d6b8968a-cee0-4cce-9ef8-b63ecaf9b815" containerName="revoke-keys"
Nov 02 20:00:30.376356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.376267    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b8968a-cee0-4cce-9ef8-b63ecaf9b815" containerName="revoke-keys"
Nov 02 20:00:30.376356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.376279    5224 cpu_manager.go:414] "containerMap: removing container" podUID="845c0067-583e-47df-a1cb-18bdaf339277" containerName="cleanup"
Nov 02 20:00:30.376356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.376284    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845c0067-583e-47df-a1cb-18bdaf339277" containerName="cleanup"
Nov 02 20:00:30.376356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.376292    5224 cpu_manager.go:414] "containerMap: removing container" podUID="c8e71803-122d-4040-8e09-f61586343778" containerName="cleanup"
Nov 02 20:00:30.376356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.376297    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e71803-122d-4040-8e09-f61586343778" containerName="cleanup"
Nov 02 20:00:30.376500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.376361    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e71803-122d-4040-8e09-f61586343778" containerName="cleanup"
Nov 02 20:00:30.376500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.376375    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b8968a-cee0-4cce-9ef8-b63ecaf9b815" containerName="revoke-keys"
Nov 02 20:00:30.379044 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.379024    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"
Nov 02 20:00:30.437845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.433582    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"]
Nov 02 20:00:30.458012 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.457967    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:00:30.458231 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.458121    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:00:30.458747 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.458726    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:00:30.458888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.458798    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:00:30.490616 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.490571    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-172adb07-r742h\" (UID: \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"
Nov 02 20:00:30.490814 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.490720    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66k5w\" (UniqueName: \"kubernetes.io/projected/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-kube-api-access-66k5w\") pod \"maas-api-revoke-keys-e2e-derive-172adb07-r742h\" (UID: \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"
Nov 02 20:00:30.490933 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.490861    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-172adb07-r742h\" (UID: \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"
Nov 02 20:00:30.503748 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.503711    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:30.591516 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.591473    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66k5w\" (UniqueName: \"kubernetes.io/projected/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-kube-api-access-66k5w\") pod \"maas-api-revoke-keys-e2e-derive-172adb07-r742h\" (UID: \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"
Nov 02 20:00:30.591704 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.591539    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-172adb07-r742h\" (UID: \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"
Nov 02 20:00:30.591704 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.591609    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-172adb07-r742h\" (UID: \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"
Nov 02 20:00:30.592250 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.592231    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-172adb07-r742h\" (UID: \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"
Nov 02 20:00:30.593938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.593920    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-172adb07-r742h\" (UID: \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"
Nov 02 20:00:30.649827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.649793    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66k5w\" (UniqueName: \"kubernetes.io/projected/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-kube-api-access-66k5w\") pod \"maas-api-revoke-keys-e2e-derive-172adb07-r742h\" (UID: \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"
Nov 02 20:00:30.691860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:30.691751    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"
Nov 02 20:00:31.016383 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:31.016351    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:31.058246 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:00:31.058211    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0a4443_f1e4_4be6_bfb4_48973a1b4939.slice/crio-c1b5767226c67e8be3869b70673c0be9459cad05f6c175c442f04829c34a15c2 WatchSource:0}: Error finding container c1b5767226c67e8be3869b70673c0be9459cad05f6c175c442f04829c34a15c2: Status 404 returned error can't find the container with id c1b5767226c67e8be3869b70673c0be9459cad05f6c175c442f04829c34a15c2
Nov 02 20:00:31.061994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:31.061960    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"]
Nov 02 20:00:31.166613 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:31.166575    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5908975c-8d31-40d2-841b-d9b88a5f483d" path="/var/lib/kubelet/pods/5908975c-8d31-40d2-841b-d9b88a5f483d/volumes"
Nov 02 20:00:31.188039 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:31.187982    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:31.366417 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:31.366378    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:31.574094 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:31.574007    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h" event={"ID":"1e0a4443-f1e4-4be6-bfb4-48973a1b4939","Type":"ContainerStarted","Data":"c8f3e01b3e09305b99247704d0140505e7bb3fce4cbf102a94dc69eb46af6e50"}
Nov 02 20:00:31.574094 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:31.574059    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h" event={"ID":"1e0a4443-f1e4-4be6-bfb4-48973a1b4939","Type":"ContainerStarted","Data":"c1b5767226c67e8be3869b70673c0be9459cad05f6c175c442f04829c34a15c2"}
Nov 02 20:00:31.690602 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:31.690530    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h" podStartSLOduration=1.690510541 podStartE2EDuration="1.690510541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:31.689232628 +0000 UTC m=+712.686921239" watchObservedRunningTime="2026-11-02 20:00:31.690510541 +0000 UTC m=+712.688199151"
Nov 02 20:00:31.847866 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:31.847747    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:32.023586 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:32.023552    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:32.202062 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:32.201961    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:32.454083 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:32.453985    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:32.580074 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:32.580035    5224 generic.go:356] "Generic (PLEG): container finished" podID="1e0a4443-f1e4-4be6-bfb4-48973a1b4939" containerID="c8f3e01b3e09305b99247704d0140505e7bb3fce4cbf102a94dc69eb46af6e50" exitCode=0
Nov 02 20:00:32.580228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:32.580123    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h" event={"ID":"1e0a4443-f1e4-4be6-bfb4-48973a1b4939","Type":"ContainerDied","Data":"c8f3e01b3e09305b99247704d0140505e7bb3fce4cbf102a94dc69eb46af6e50"}
Nov 02 20:00:33.056403 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:33.056366    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:00:33.720163 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:33.720136    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"
Nov 02 20:00:33.820675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:33.820640    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-xks-ca\") pod \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\" (UID: \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\") "
Nov 02 20:00:33.820901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:33.820711    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-openshift-ca\") pod \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\" (UID: \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\") "
Nov 02 20:00:33.820901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:33.820802    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-kube-api-access-66k5w\" (UniqueName: \"kubernetes.io/projected/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-kube-api-access-66k5w\") pod \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\" (UID: \"1e0a4443-f1e4-4be6-bfb4-48973a1b4939\") "
Nov 02 20:00:33.821085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:33.821058    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-openshift-ca" pod "1e0a4443-f1e4-4be6-bfb4-48973a1b4939" (UID: "1e0a4443-f1e4-4be6-bfb4-48973a1b4939"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:00:33.822973 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:33.822941    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-xks-ca" pod "1e0a4443-f1e4-4be6-bfb4-48973a1b4939" (UID: "1e0a4443-f1e4-4be6-bfb4-48973a1b4939"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:00:33.823073 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:33.822947    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-kube-api-access-66k5w" pod "1e0a4443-f1e4-4be6-bfb4-48973a1b4939" (UID: "1e0a4443-f1e4-4be6-bfb4-48973a1b4939"). InnerVolumeSpecName "kube-api-access-66k5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:00:33.922431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:33.922342    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:33.922431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:33.922378    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:33.922431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:33.922388    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66k5w\" (UniqueName: \"kubernetes.io/projected/1e0a4443-f1e4-4be6-bfb4-48973a1b4939-kube-api-access-66k5w\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:00:34.049920 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:34.049889    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:34.221928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:34.221836    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:34.591260 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:34.591214    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h" event={"ID":"1e0a4443-f1e4-4be6-bfb4-48973a1b4939","Type":"ContainerDied","Data":"c1b5767226c67e8be3869b70673c0be9459cad05f6c175c442f04829c34a15c2"}
Nov 02 20:00:34.591260 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:34.591243    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"
Nov 02 20:00:34.591489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:34.591314    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1b5767226c67e8be3869b70673c0be9459cad05f6c175c442f04829c34a15c2"
Nov 02 20:00:34.908904 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:34.908817    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:35.097538 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:35.097498    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:35.279763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:35.279723    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:35.867444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:35.867405    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:36.410730 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:36.410697    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:36.595046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:36.595012    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:37.850093 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:37.850056    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:38.119544 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:38.119462    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:38.305073 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:38.305034    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:39.244191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:39.244158    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:39.487043 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:39.486950    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:39.672664 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:39.672625    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:39.854149 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:39.854096    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:42.733211 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:42.733171    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:42.906689 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:42.906655    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:43.113121 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:43.113089    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:43.396638 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:43.396564    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:43.709909 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:43.709819    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:43.955957 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:43.955917    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"]
Nov 02 20:00:44.007137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:44.007046    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3acf2229-vl7lx"]
Nov 02 20:00:45.166424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:45.166389    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b8968a-cee0-4cce-9ef8-b63ecaf9b815" path="/var/lib/kubelet/pods/d6b8968a-cee0-4cce-9ef8-b63ecaf9b815/volumes"
Nov 02 20:00:45.442082 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:45.441977    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:46.018923 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:46.018894    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:46.243797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:46.243740    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:46.823918 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:46.823886    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:47.073761 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:47.073720    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:47.677024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:47.676991    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:00:48.216414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:48.216335    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:48.442897 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:48.442839    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:48.734015 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:48.733978    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:49.477724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:49.477687    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:50.581115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:50.581080    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:50.803487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:50.803451    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:52.374460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:52.374432    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:52.606202 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:52.606167    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:53.687271 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:53.687227    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:53.867532 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:53.867490    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:54.045794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:54.045737    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:57.358494 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:00:57.358449    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:00:58.887534 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:58.887488    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:59.061458 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:59.061420    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:59.233515 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:59.233436    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:59.538701 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:59.538666    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:00:59.713374 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:00:59.713334    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:03.327039 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:03.326999    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:04.653145 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:04.653114    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:01:04.951613 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:04.951526    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:05.262139 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:05.262060    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:05.429269 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:05.429237    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"]
Nov 02 20:01:05.449333 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:05.449299    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:05.482620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:05.482588    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-172adb07-r742h"]
Nov 02 20:01:05.640062 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:05.640000    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:06.395825 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:06.395782    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:07.018507 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:07.018473    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:07.166611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:07.166577    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0a4443-f1e4-4be6-bfb4-48973a1b4939" path="/var/lib/kubelet/pods/1e0a4443-f1e4-4be6-bfb4-48973a1b4939/volumes"
Nov 02 20:01:07.438508 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:07.438414    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:10.678555 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:10.678510    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:10.938176 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:10.938071    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:11.145677 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:11.145641    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:11.335881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:11.335848    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:12.761506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:12.761472    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:01:14.137562 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.137527    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:14.359413 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.359378    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:14.502466 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.502381    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"]
Nov 02 20:01:14.502818 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.502804    5224 cpu_manager.go:414] "containerMap: removing container" podUID="1e0a4443-f1e4-4be6-bfb4-48973a1b4939" containerName="revoke-keys"
Nov 02 20:01:14.502865 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.502820    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0a4443-f1e4-4be6-bfb4-48973a1b4939" containerName="revoke-keys"
Nov 02 20:01:14.502899 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.502883    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0a4443-f1e4-4be6-bfb4-48973a1b4939" containerName="revoke-keys"
Nov 02 20:01:14.506167 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.506152    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"
Nov 02 20:01:14.563069 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.563035    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"]
Nov 02 20:01:14.609610 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.609576    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/509e4240-8ed0-4980-90bc-53c5c21ffea8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t\" (UID: \"509e4240-8ed0-4980-90bc-53c5c21ffea8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"
Nov 02 20:01:14.609810 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.609613    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvr8z\" (UniqueName: \"kubernetes.io/projected/509e4240-8ed0-4980-90bc-53c5c21ffea8-kube-api-access-vvr8z\") pod \"payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t\" (UID: \"509e4240-8ed0-4980-90bc-53c5c21ffea8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"
Nov 02 20:01:14.611365 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.611348    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-bbee67\""
Nov 02 20:01:14.611414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.611374    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-bbee67-dockercfg-xscq4\""
Nov 02 20:01:14.710246 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.710207    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/509e4240-8ed0-4980-90bc-53c5c21ffea8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t\" (UID: \"509e4240-8ed0-4980-90bc-53c5c21ffea8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"
Nov 02 20:01:14.710246 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.710247    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvr8z\" (UniqueName: \"kubernetes.io/projected/509e4240-8ed0-4980-90bc-53c5c21ffea8-kube-api-access-vvr8z\") pod \"payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t\" (UID: \"509e4240-8ed0-4980-90bc-53c5c21ffea8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"
Nov 02 20:01:14.710927 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.710909    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/509e4240-8ed0-4980-90bc-53c5c21ffea8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t\" (UID: \"509e4240-8ed0-4980-90bc-53c5c21ffea8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"
Nov 02 20:01:14.768455 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.768423    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvr8z\" (UniqueName: \"kubernetes.io/projected/509e4240-8ed0-4980-90bc-53c5c21ffea8-kube-api-access-vvr8z\") pod \"payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t\" (UID: \"509e4240-8ed0-4980-90bc-53c5c21ffea8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"
Nov 02 20:01:14.818669 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:14.818637    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"
Nov 02 20:01:15.184944 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:01:15.184909    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509e4240_8ed0_4980_90bc_53c5c21ffea8.slice/crio-e64fbc84e010b82653a3b91cbb085b9d8637902f905384c99b9c0eb6bf422c6e WatchSource:0}: Error finding container e64fbc84e010b82653a3b91cbb085b9d8637902f905384c99b9c0eb6bf422c6e: Status 404 returned error can't find the container with id e64fbc84e010b82653a3b91cbb085b9d8637902f905384c99b9c0eb6bf422c6e
Nov 02 20:01:15.187385 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:15.187360    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"]
Nov 02 20:01:15.792409 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:15.792364    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t" event={"ID":"509e4240-8ed0-4980-90bc-53c5c21ffea8","Type":"ContainerStarted","Data":"e64fbc84e010b82653a3b91cbb085b9d8637902f905384c99b9c0eb6bf422c6e"}
Nov 02 20:01:15.893421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:15.893383    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:16.066060 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:16.065963    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:17.803785 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:17.803739    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t" event={"ID":"509e4240-8ed0-4980-90bc-53c5c21ffea8","Type":"ContainerStarted","Data":"767c1f960ad5deb32bcff979ca10eaab72e2ff5d598f879b7edab76f00fe0310"}
Nov 02 20:01:17.804120 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:17.803834    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"
Nov 02 20:01:17.920434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:17.920385    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t" podStartSLOduration=1.400140627 podStartE2EDuration="3.920368675s" totalImagesPullingTime="2.520228048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:17.919657067 +0000 UTC m=+758.917345675" watchObservedRunningTime="2026-11-02 20:01:17.920368675 +0000 UTC m=+758.918057285"
Nov 02 20:01:20.298137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:20.298092    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:20.890606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:20.890571    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:21.199516 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:21.199415    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:21.734680 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:21.734641    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:24.126716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.126676    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"]
Nov 02 20:01:24.129501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.129477    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.200085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.200045    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-tmp-dir\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.200272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.200093    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67432d1-22af-458b-bd57-002a02c62a5d-tls-certs\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.200272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.200192    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-home\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.200272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.200249    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-model-cache\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.200460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.200308    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-dshm\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.200460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.200362    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9995f\" (UniqueName: \"kubernetes.io/projected/f67432d1-22af-458b-bd57-002a02c62a5d-kube-api-access-9995f\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.235123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.235086    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-426941\"/\"kube-root-ca.crt\""
Nov 02 20:01:24.235286 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.235154    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-426941\"/\"openshift-service-ca.crt\""
Nov 02 20:01:24.235697 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.235677    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-426941\"/\"default-dockercfg-hxc6x\""
Nov 02 20:01:24.235697 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.235689    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-426941\"/\"simulator-w2-426941-kserve-self-signed-certs\""
Nov 02 20:01:24.236565 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.236545    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"]
Nov 02 20:01:24.269065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.269032    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:24.301338 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.301296    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-model-cache\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.301527 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.301345    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-dshm\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.301527 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.301397    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9995f\" (UniqueName: \"kubernetes.io/projected/f67432d1-22af-458b-bd57-002a02c62a5d-kube-api-access-9995f\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.301527 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.301449    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-tmp-dir\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.301527 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.301501    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67432d1-22af-458b-bd57-002a02c62a5d-tls-certs\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.301757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.301564    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-home\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.301757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.301732    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-model-cache\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.302827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.301931    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-home\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.302827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.302095    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-tmp-dir\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.304471 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.304451    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-dshm\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.305107 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.305086    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67432d1-22af-458b-bd57-002a02c62a5d-tls-certs\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.359910 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.359872    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9995f\" (UniqueName: \"kubernetes.io/projected/f67432d1-22af-458b-bd57-002a02c62a5d-kube-api-access-9995f\") pod \"simulator-w2-426941-kserve-7795cc9fdf-gvdg6\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") " pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.440568 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.440471    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:24.807392 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:01:24.807360    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf67432d1_22af_458b_bd57_002a02c62a5d.slice/crio-78354821ec14decc0d7ff2ec9a9d627c4ed67a4e5ff7d20bd2a896e1f1f12f0b WatchSource:0}: Error finding container 78354821ec14decc0d7ff2ec9a9d627c4ed67a4e5ff7d20bd2a896e1f1f12f0b: Status 404 returned error can't find the container with id 78354821ec14decc0d7ff2ec9a9d627c4ed67a4e5ff7d20bd2a896e1f1f12f0b
Nov 02 20:01:24.809291 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.809271    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"]
Nov 02 20:01:24.840611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:24.840580    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6" event={"ID":"f67432d1-22af-458b-bd57-002a02c62a5d","Type":"ContainerStarted","Data":"78354821ec14decc0d7ff2ec9a9d627c4ed67a4e5ff7d20bd2a896e1f1f12f0b"}
Nov 02 20:01:27.191619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:27.191589    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:27.367303 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:01:27.367264    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:01:27.870216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:27.870175    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6" event={"ID":"f67432d1-22af-458b-bd57-002a02c62a5d","Type":"ContainerStarted","Data":"1d8bddd2e248539e1af43882fb53681d9dd509f9570ec2e3cef6b6b0f42586dd"}
Nov 02 20:01:27.988758 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:27.988650    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6" podStartSLOduration=1.14282594 podStartE2EDuration="3.988634787s" totalImagesPullingTime="2.845808847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:27.987432267 +0000 UTC m=+768.985120892" watchObservedRunningTime="2026-11-02 20:01:27.988634787 +0000 UTC m=+768.986323395"
Nov 02 20:01:28.441592 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:28.441556    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:28.812723 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:28.812693    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"
Nov 02 20:01:30.281003 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:30.280969    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:01:32.278688 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:32.278652    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:32.808724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:32.808689    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:33.008009 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:33.007967    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:33.195566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:33.195476    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:33.378892 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:33.378861    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:33.945153 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:33.945118    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:34.133846 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:34.133800    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:34.440872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:34.440840    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:34.453316 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:34.453293    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:34.921846 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:34.921812    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:01:38.140902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:38.140860    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:38.340760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:38.340723    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:38.535217 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:38.535180    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:38.720406 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:38.720371    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:44.328454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:44.328422    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:44.865659 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:44.865612    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:01:48.360522 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:48.360471    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:49.055695 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.055660    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:49.548301 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.548267    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"]
Nov 02 20:01:49.554107 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.554083    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.605016 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.604986    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"]
Nov 02 20:01:49.643301 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.643267    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-home\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.643449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.643309    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-dshm\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.643449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.643381    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41edc495-0352-461a-a80c-c4dd7358e0c3-tls-certs\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.643449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.643411    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-model-cache\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.643449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.643444    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmb7\" (UniqueName: \"kubernetes.io/projected/41edc495-0352-461a-a80c-c4dd7358e0c3-kube-api-access-qlmb7\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.643593 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.643572    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-tmp-dir\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.654761 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.654731    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-14d090\"/\"default-dockercfg-xfwx7\""
Nov 02 20:01:49.654761 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.654729    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-14d090\"/\"openshift-service-ca.crt\""
Nov 02 20:01:49.654957 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.654835    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-14d090\"/\"kube-root-ca.crt\""
Nov 02 20:01:49.658820 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.658805    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-14d090\"/\"simulator-w0-14d090-kserve-self-signed-certs\""
Nov 02 20:01:49.684727 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.684691    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:49.744659 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.744614    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41edc495-0352-461a-a80c-c4dd7358e0c3-tls-certs\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.744835 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.744671    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-model-cache\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.744835 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.744726    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmb7\" (UniqueName: \"kubernetes.io/projected/41edc495-0352-461a-a80c-c4dd7358e0c3-kube-api-access-qlmb7\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.744835 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.744818    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-tmp-dir\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.745019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.744909    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-home\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.745019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.744953    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-dshm\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.745195 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.745162    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-model-cache\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.745329 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.745276    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-tmp-dir\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.745387 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.745363    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-home\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.747299 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.747271    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-dshm\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.747456 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.747439    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41edc495-0352-461a-a80c-c4dd7358e0c3-tls-certs\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.804513 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.804432    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmb7\" (UniqueName: \"kubernetes.io/projected/41edc495-0352-461a-a80c-c4dd7358e0c3-kube-api-access-qlmb7\") pod \"simulator-w0-14d090-kserve-598df86587-9n6gk\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") " pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:49.862292 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.862255    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:49.864119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:49.864092    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:50.234496 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:01:50.234464    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41edc495_0352_461a_a80c_c4dd7358e0c3.slice/crio-27e40820862f773bc32e0ff08b54b517ef1aff9e9f86b0fd4034fbaa81e29ac0 WatchSource:0}: Error finding container 27e40820862f773bc32e0ff08b54b517ef1aff9e9f86b0fd4034fbaa81e29ac0: Status 404 returned error can't find the container with id 27e40820862f773bc32e0ff08b54b517ef1aff9e9f86b0fd4034fbaa81e29ac0
Nov 02 20:01:50.237113 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:50.237090    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"]
Nov 02 20:01:50.992371 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:50.992337    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk" event={"ID":"41edc495-0352-461a-a80c-c4dd7358e0c3","Type":"ContainerStarted","Data":"9a031f02161244c7504fb57b2a0b440444f6a722b8e8346fb5d98ae5ef3a2805"}
Nov 02 20:01:50.992371 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:50.992378    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk" event={"ID":"41edc495-0352-461a-a80c-c4dd7358e0c3","Type":"ContainerStarted","Data":"27e40820862f773bc32e0ff08b54b517ef1aff9e9f86b0fd4034fbaa81e29ac0"}
Nov 02 20:01:51.109451 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:51.109367    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk" podStartSLOduration=2.109354472 podStartE2EDuration="2.109354472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:51.108464454 +0000 UTC m=+792.106153076" watchObservedRunningTime="2026-11-02 20:01:51.109354472 +0000 UTC m=+792.107043078"
Nov 02 20:01:51.864863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:51.864823    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:53.173696 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:53.173658    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:01:53.260337 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:53.260300    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:57.070415 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:57.070371    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:57.357244 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:01:57.357158    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:01:57.651923 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:57.651837    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:57.885422 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:57.885392    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:58.589000 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.588965    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:58.710162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.710133    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"]
Nov 02 20:01:58.712830 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.712813    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.766238 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.766208    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"]
Nov 02 20:01:58.777599 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.777577    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6a4ab0\"/\"openshift-service-ca.crt\""
Nov 02 20:01:58.777759 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.777601    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6a4ab0\"/\"kube-root-ca.crt\""
Nov 02 20:01:58.799384 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.799354    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6a4ab0\"/\"default-dockercfg-f8g54\""
Nov 02 20:01:58.799529 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.799439    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6a4ab0\"/\"simulator-w3-6a4ab0-kserve-self-signed-certs\""
Nov 02 20:01:58.828600 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.828571    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-dshm\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.828600 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.828603    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcc6s\" (UniqueName: \"kubernetes.io/projected/aa503409-81d4-4532-9dc8-64a7aa8d0443-kube-api-access-pcc6s\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.828847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.828719    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-home\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.828847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.828759    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-model-cache\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.828847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.828810    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa503409-81d4-4532-9dc8-64a7aa8d0443-tls-certs\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.828949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.828896    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-tmp-dir\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.837870 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.837843    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:01:58.930176 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.930096    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-dshm\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.930176 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.930141    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcc6s\" (UniqueName: \"kubernetes.io/projected/aa503409-81d4-4532-9dc8-64a7aa8d0443-kube-api-access-pcc6s\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.930397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.930223    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-home\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.930397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.930260    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-model-cache\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.930397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.930286    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa503409-81d4-4532-9dc8-64a7aa8d0443-tls-certs\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.930397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.930374    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-tmp-dir\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.930718 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.930692    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-model-cache\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.930804 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.930739    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-tmp-dir\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.930955 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.930930    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-home\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.932454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.932431    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-dshm\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.933009 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.932979    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa503409-81d4-4532-9dc8-64a7aa8d0443-tls-certs\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:58.988983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:58.988954    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcc6s\" (UniqueName: \"kubernetes.io/projected/aa503409-81d4-4532-9dc8-64a7aa8d0443-kube-api-access-pcc6s\") pod \"simulator-w3-6a4ab0-kserve-756bd89d98-brcng\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") " pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:59.028054 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:59.028027    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:01:59.398257 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:01:59.398228    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa503409_81d4_4532_9dc8_64a7aa8d0443.slice/crio-50e7d4a17d0d0801e4ce605fd1fe88c280f2114f34481d9af455368ddb0a2d45 WatchSource:0}: Error finding container 50e7d4a17d0d0801e4ce605fd1fe88c280f2114f34481d9af455368ddb0a2d45: Status 404 returned error can't find the container with id 50e7d4a17d0d0801e4ce605fd1fe88c280f2114f34481d9af455368ddb0a2d45
Nov 02 20:01:59.400913 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:59.400796    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"]
Nov 02 20:01:59.864540 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:59.864505    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:01:59.878074 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:01:59.878044    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:02:00.041134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:00.041102    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" event={"ID":"aa503409-81d4-4532-9dc8-64a7aa8d0443","Type":"ContainerStarted","Data":"3a34d3737cd7f163cfd09a3f2f0adacecbded81af8fd2a27232e97bb134c887e"}
Nov 02 20:02:00.041311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:00.041143    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" event={"ID":"aa503409-81d4-4532-9dc8-64a7aa8d0443","Type":"ContainerStarted","Data":"50e7d4a17d0d0801e4ce605fd1fe88c280f2114f34481d9af455368ddb0a2d45"}
Nov 02 20:02:00.052172 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:00.052145    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:02:00.157753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:00.157672    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" podStartSLOduration=2.157658663 podStartE2EDuration="2.157658663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:00.157138964 +0000 UTC m=+801.154827572" watchObservedRunningTime="2026-11-02 20:02:00.157658663 +0000 UTC m=+801.155347270"
Nov 02 20:02:01.028200 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:01.028166    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:02:03.553027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:03.552982    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:07.286356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:07.286318    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:08.560472 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:08.560426    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:09.028789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:09.028743    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:02:09.041309 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:09.041286    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:02:09.099308 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:09.099285    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:02:13.153991 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:13.153959    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:02:13.524136 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:13.524101    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:16.933415 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:16.933376    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:17.554404 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:17.554369    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:17.756195 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:17.756149    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:18.683457 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:18.683359    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:22.905738 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:22.905705    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"]
Nov 02 20:02:22.908004 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:22.907988    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.015178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.015142    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c25349\"/\"openshift-service-ca.crt\""
Nov 02 20:02:23.015178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.015154    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c25349\"/\"kube-root-ca.crt\""
Nov 02 20:02:23.015402 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.015224    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c25349\"/\"default-dockercfg-hn4wb\""
Nov 02 20:02:23.015402 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.015302    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c25349\"/\"simulator-w4-c25349-kserve-self-signed-certs\""
Nov 02 20:02:23.017075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.017056    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"]
Nov 02 20:02:23.034930 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.034898    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:23.071748 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.071711    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tls-certs\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.072021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.071850    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmn27\" (UniqueName: \"kubernetes.io/projected/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-kube-api-access-nmn27\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.072021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.071902    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-dshm\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.072170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.071997    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-model-cache\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.072170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.072153    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-home\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.072327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.072278    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tmp-dir\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.173396 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.173312    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tmp-dir\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.173396 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.173383    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tls-certs\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.173643 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.173443    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmn27\" (UniqueName: \"kubernetes.io/projected/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-kube-api-access-nmn27\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.173643 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.173475    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-dshm\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.173643 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.173501    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-model-cache\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.173643 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.173532    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-home\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.173888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.173844    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tmp-dir\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.173984 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.173960    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-model-cache\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.174191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.174168    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-home\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.175858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.175833    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-dshm\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.176220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.176199    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tls-certs\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.231555 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.231519    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmn27\" (UniqueName: \"kubernetes.io/projected/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-kube-api-access-nmn27\") pod \"simulator-w4-c25349-kserve-5dbc9cfffc-bhptj\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") " pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.518467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.518434    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:23.589940 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.589829    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:23.882458 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:02:23.882426    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4ff50fb_2482_4330_9ce4_e05d794c2a2a.slice/crio-356839997d8b307a616aed23b10a5def9ec333e0c8332e59efdca22e0338ed55 WatchSource:0}: Error finding container 356839997d8b307a616aed23b10a5def9ec333e0c8332e59efdca22e0338ed55: Status 404 returned error can't find the container with id 356839997d8b307a616aed23b10a5def9ec333e0c8332e59efdca22e0338ed55
Nov 02 20:02:23.884973 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:23.884952    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"]
Nov 02 20:02:24.169801 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:24.169699    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj" event={"ID":"f4ff50fb-2482-4330-9ce4-e05d794c2a2a","Type":"ContainerStarted","Data":"181de45ea58399596f802acae56c7ef80189d2f6249bb9bf00be5388ba9ffabc"}
Nov 02 20:02:24.169801 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:24.169747    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj" event={"ID":"f4ff50fb-2482-4330-9ce4-e05d794c2a2a","Type":"ContainerStarted","Data":"356839997d8b307a616aed23b10a5def9ec333e0c8332e59efdca22e0338ed55"}
Nov 02 20:02:24.295176 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:24.291133    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj" podStartSLOduration=2.291114889 podStartE2EDuration="2.291114889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:24.287302947 +0000 UTC m=+825.284991557" watchObservedRunningTime="2026-11-02 20:02:24.291114889 +0000 UTC m=+825.288803499"
Nov 02 20:02:24.519260 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:24.519225    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:27.357846 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:02:27.357814    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:02:28.459327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:28.459291    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:31.133272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:31.133236    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:02:33.308587 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:33.308538    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:33.519558 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:33.519517    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:33.532483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:33.532455    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:34.228743 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:34.228713    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:02:38.265354 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:38.265313    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:40.484439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:40.484390    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:43.166281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:43.166247    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:44.668956 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:44.668921    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:02:48.071295 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:48.071257    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:49.229355 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.229323    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"]
Nov 02 20:02:49.231847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.231830    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.285265 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.285227    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"]
Nov 02 20:02:49.326283 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.326247    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-dshm\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.326459 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.326292    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4grtv\" (UniqueName: \"kubernetes.io/projected/675c9473-71e0-4bd1-8804-e684edf052fb-kube-api-access-4grtv\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.326515 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.326476    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-tmp-dir\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.326602 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.326579    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-model-cache\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.326662 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.326624    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-home\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.326728 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.326702    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/675c9473-71e0-4bd1-8804-e684edf052fb-tls-certs\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.335210 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.335186    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"simulator-w1-bbee67-kserve-self-signed-certs\""
Nov 02 20:02:49.335345 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.335249    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"default-dockercfg-gj7fz\""
Nov 02 20:02:49.336678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.336660    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"openshift-service-ca.crt\""
Nov 02 20:02:49.336722 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.336660    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"kube-root-ca.crt\""
Nov 02 20:02:49.389140 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.389108    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:49.428151 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.428118    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-tmp-dir\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.428151 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.428157    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-model-cache\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.428439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.428182    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-home\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.428439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.428301    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/675c9473-71e0-4bd1-8804-e684edf052fb-tls-certs\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.428439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.428348    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-dshm\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.428439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.428398    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4grtv\" (UniqueName: \"kubernetes.io/projected/675c9473-71e0-4bd1-8804-e684edf052fb-kube-api-access-4grtv\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.428619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.428551    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-model-cache\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.428668 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.428624    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-tmp-dir\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.428668 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.428635    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-home\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.430616 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.430591    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-dshm\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.430890 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.430872    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/675c9473-71e0-4bd1-8804-e684edf052fb-tls-certs\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.486977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.486908    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4grtv\" (UniqueName: \"kubernetes.io/projected/675c9473-71e0-4bd1-8804-e684edf052fb-kube-api-access-4grtv\") pod \"simulator-w1-bbee67-kserve-d9c686c9-gjgq6\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") " pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.542542 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.542502    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:49.568003 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.567968    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:49.912997 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:02:49.912929    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod675c9473_71e0_4bd1_8804_e684edf052fb.slice/crio-aa70efff6531fe8b30859c1cdfccd8997baf625b62e657e1583f4ba95a051c91 WatchSource:0}: Error finding container aa70efff6531fe8b30859c1cdfccd8997baf625b62e657e1583f4ba95a051c91: Status 404 returned error can't find the container with id aa70efff6531fe8b30859c1cdfccd8997baf625b62e657e1583f4ba95a051c91
Nov 02 20:02:49.917049 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:49.917017    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"]
Nov 02 20:02:50.309944 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:50.309901    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6" event={"ID":"675c9473-71e0-4bd1-8804-e684edf052fb","Type":"ContainerStarted","Data":"a386d8ce57d4281fd0aef3867beb5f7342048fc6c771da4a27cbc43b15e59125"}
Nov 02 20:02:50.309944 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:50.309948    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6" event={"ID":"675c9473-71e0-4bd1-8804-e684edf052fb","Type":"ContainerStarted","Data":"aa70efff6531fe8b30859c1cdfccd8997baf625b62e657e1583f4ba95a051c91"}
Nov 02 20:02:50.426836 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:50.426791    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6" podStartSLOduration=1.426763266 podStartE2EDuration="1.426763266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:50.425840221 +0000 UTC m=+851.423528829" watchObservedRunningTime="2026-11-02 20:02:50.426763266 +0000 UTC m=+851.424451875"
Nov 02 20:02:50.447363 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:50.447321    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:50.542987 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:50.542940    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:52.972273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:52.972233    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:57.358124 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:02:57.358088    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:02:57.846643 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:57.846611    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:02:59.542713 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:59.542675    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:02:59.561309 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:02:59.561273    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:03:00.372757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:00.372729    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:03:01.191390 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:01.189723    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:02.755354 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:02.755313    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:04.658216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.658183    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"]
Nov 02 20:03:04.660739 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.660721    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.676037 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.676005    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp8dc\" (UniqueName: \"kubernetes.io/projected/87bb590b-e94c-4051-9df7-09be7673c1e8-kube-api-access-qp8dc\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.676157 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.676052    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-dshm\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.676199 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.676161    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-home\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.676199 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.676187    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-tmp-dir\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.676298 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.676210    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87bb590b-e94c-4051-9df7-09be7673c1e8-tls-certs\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.676298 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.676265    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-model-cache\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.714481 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.714447    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"]
Nov 02 20:03:04.764196 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.764168    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6a4ab0\"/\"premium-w3-6a4ab0-kserve-self-signed-certs\""
Nov 02 20:03:04.776799 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.776736    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp8dc\" (UniqueName: \"kubernetes.io/projected/87bb590b-e94c-4051-9df7-09be7673c1e8-kube-api-access-qp8dc\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.776962 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.776807    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-dshm\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.776962 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.776855    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-home\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.776962 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.776870    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-tmp-dir\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.776962 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.776894    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87bb590b-e94c-4051-9df7-09be7673c1e8-tls-certs\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.776962 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.776928    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-model-cache\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.777372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.777340    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-home\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.777372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.777360    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-model-cache\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.777541 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.777443    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-tmp-dir\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.779302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.779285    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-dshm\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.779588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.779566    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87bb590b-e94c-4051-9df7-09be7673c1e8-tls-certs\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.793462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.793430    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:04.849397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.849361    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp8dc\" (UniqueName: \"kubernetes.io/projected/87bb590b-e94c-4051-9df7-09be7673c1e8-kube-api-access-qp8dc\") pod \"premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") " pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:04.972038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:04.971942    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:05.348899 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:03:05.348857    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87bb590b_e94c_4051_9df7_09be7673c1e8.slice/crio-d2c0c8335f1164328d1c1112ca442c69a75fc7012d09ad1505bd2c43edee2565 WatchSource:0}: Error finding container d2c0c8335f1164328d1c1112ca442c69a75fc7012d09ad1505bd2c43edee2565: Status 404 returned error can't find the container with id d2c0c8335f1164328d1c1112ca442c69a75fc7012d09ad1505bd2c43edee2565
Nov 02 20:03:05.350169 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:05.350141    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"]
Nov 02 20:03:05.389645 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:05.389611    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" event={"ID":"87bb590b-e94c-4051-9df7-09be7673c1e8","Type":"ContainerStarted","Data":"d2c0c8335f1164328d1c1112ca442c69a75fc7012d09ad1505bd2c43edee2565"}
Nov 02 20:03:06.395761 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:06.395726    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" event={"ID":"87bb590b-e94c-4051-9df7-09be7673c1e8","Type":"ContainerStarted","Data":"8502226f1e020a1490dd1690cc7cf28e553c91912279059b608110e2e278d167"}
Nov 02 20:03:06.513450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:06.513397    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" podStartSLOduration=2.51338311 podStartE2EDuration="2.51338311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:06.511935717 +0000 UTC m=+867.509624326" watchObservedRunningTime="2026-11-02 20:03:06.51338311 +0000 UTC m=+867.511071761"
Nov 02 20:03:06.676539 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:06.676445    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:03:06.972262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:06.972178    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:07.673583 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:07.673543    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:12.556351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:12.556308    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:14.972989 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:14.972945    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:14.985265 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:14.985236    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:15.454271 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:15.454240    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:03:17.553281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:17.553240    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:22.477501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:22.477445    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:24.704035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:24.704000    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:03:27.358316 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:03:27.358265    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:03:27.469388 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:27.469335    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:31.417405 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:31.417364    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:32.426415 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:32.426375    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:34.814026 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:34.813983    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:35.015471 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:35.015435    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:37.375311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:37.375245    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:42.274582 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:42.274544    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:47.298426 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:47.298390    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:49.767569 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:49.767496    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:50.457212 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:50.457176    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:52.149913 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:52.149866    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:52.998710 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:52.998670    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:57.062385 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:57.062348    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:03:57.357276 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:03:57.357194    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:03:58.494159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:03:58.494128    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:04:01.961567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:01.961530    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:05.056077 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.056045    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"]
Nov 02 20:04:05.059676 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.059660    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"
Nov 02 20:04:05.112402 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.112366    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"]
Nov 02 20:04:05.162010 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.161977    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:04:05.162010 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.161986    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:04:05.162220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.162023    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:04:05.162220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.162076    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:04:05.162220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.162072    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c86ff75-rcj8r\" (UID: \"deceea06-2be9-48c0-b59e-68e149119bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"
Nov 02 20:04:05.162220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.162149    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c86ff75-rcj8r\" (UID: \"deceea06-2be9-48c0-b59e-68e149119bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"
Nov 02 20:04:05.162220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.162188    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slhvv\" (UniqueName: \"kubernetes.io/projected/deceea06-2be9-48c0-b59e-68e149119bed-kube-api-access-slhvv\") pod \"maas-api-revoke-sub-9c86ff75-rcj8r\" (UID: \"deceea06-2be9-48c0-b59e-68e149119bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"
Nov 02 20:04:05.263466 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.263427    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c86ff75-rcj8r\" (UID: \"deceea06-2be9-48c0-b59e-68e149119bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"
Nov 02 20:04:05.263466 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.263471    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c86ff75-rcj8r\" (UID: \"deceea06-2be9-48c0-b59e-68e149119bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"
Nov 02 20:04:05.263693 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.263493    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slhvv\" (UniqueName: \"kubernetes.io/projected/deceea06-2be9-48c0-b59e-68e149119bed-kube-api-access-slhvv\") pod \"maas-api-revoke-sub-9c86ff75-rcj8r\" (UID: \"deceea06-2be9-48c0-b59e-68e149119bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"
Nov 02 20:04:05.264228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.264209    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c86ff75-rcj8r\" (UID: \"deceea06-2be9-48c0-b59e-68e149119bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"
Nov 02 20:04:05.265788 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.265757    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c86ff75-rcj8r\" (UID: \"deceea06-2be9-48c0-b59e-68e149119bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"
Nov 02 20:04:05.321646 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.321579    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slhvv\" (UniqueName: \"kubernetes.io/projected/deceea06-2be9-48c0-b59e-68e149119bed-kube-api-access-slhvv\") pod \"maas-api-revoke-sub-9c86ff75-rcj8r\" (UID: \"deceea06-2be9-48c0-b59e-68e149119bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"
Nov 02 20:04:05.371962 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.371934    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"
Nov 02 20:04:05.734129 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:04:05.734098    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeceea06_2be9_48c0_b59e_68e149119bed.slice/crio-2763f20a91b59e8b3b41928bc95a38ce6ed877f5a7423253ad208cb33df0116a WatchSource:0}: Error finding container 2763f20a91b59e8b3b41928bc95a38ce6ed877f5a7423253ad208cb33df0116a: Status 404 returned error can't find the container with id 2763f20a91b59e8b3b41928bc95a38ce6ed877f5a7423253ad208cb33df0116a
Nov 02 20:04:05.737300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:05.737277    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"]
Nov 02 20:04:06.710433 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:06.710398    5224 generic.go:356] "Generic (PLEG): container finished" podID="deceea06-2be9-48c0-b59e-68e149119bed" containerID="46b0dcba66bfc488ada20b9ef6d34514940f91e94192b2038e30f5fe1bad77fd" exitCode=0
Nov 02 20:04:06.710828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:06.710438    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r" event={"ID":"deceea06-2be9-48c0-b59e-68e149119bed","Type":"ContainerDied","Data":"46b0dcba66bfc488ada20b9ef6d34514940f91e94192b2038e30f5fe1bad77fd"}
Nov 02 20:04:06.710828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:06.710496    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r" event={"ID":"deceea06-2be9-48c0-b59e-68e149119bed","Type":"ContainerStarted","Data":"2763f20a91b59e8b3b41928bc95a38ce6ed877f5a7423253ad208cb33df0116a"}
Nov 02 20:04:07.174900 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:07.174869    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:07.851495 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:07.851467    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"
Nov 02 20:04:07.991909 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:07.991833    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deceea06-2be9-48c0-b59e-68e149119bed-kube-api-access-slhvv\" (UniqueName: \"kubernetes.io/projected/deceea06-2be9-48c0-b59e-68e149119bed-kube-api-access-slhvv\") pod \"deceea06-2be9-48c0-b59e-68e149119bed\" (UID: \"deceea06-2be9-48c0-b59e-68e149119bed\") "
Nov 02 20:04:07.991909 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:07.991872    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-xks-ca\") pod \"deceea06-2be9-48c0-b59e-68e149119bed\" (UID: \"deceea06-2be9-48c0-b59e-68e149119bed\") "
Nov 02 20:04:07.991909 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:07.991906    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-openshift-ca\") pod \"deceea06-2be9-48c0-b59e-68e149119bed\" (UID: \"deceea06-2be9-48c0-b59e-68e149119bed\") "
Nov 02 20:04:07.992307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:07.992282    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-openshift-ca" pod "deceea06-2be9-48c0-b59e-68e149119bed" (UID: "deceea06-2be9-48c0-b59e-68e149119bed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:04:07.994036 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:07.994005    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deceea06-2be9-48c0-b59e-68e149119bed-kube-api-access-slhvv" pod "deceea06-2be9-48c0-b59e-68e149119bed" (UID: "deceea06-2be9-48c0-b59e-68e149119bed"). InnerVolumeSpecName "kube-api-access-slhvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:04:07.994141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:07.994032    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-xks-ca" pod "deceea06-2be9-48c0-b59e-68e149119bed" (UID: "deceea06-2be9-48c0-b59e-68e149119bed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:04:08.093384 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:08.093353    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slhvv\" (UniqueName: \"kubernetes.io/projected/deceea06-2be9-48c0-b59e-68e149119bed-kube-api-access-slhvv\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:08.093384 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:08.093380    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:08.093384 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:08.093389    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deceea06-2be9-48c0-b59e-68e149119bed-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:04:08.721940 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:08.721911    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"
Nov 02 20:04:08.722119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:08.721904    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r" event={"ID":"deceea06-2be9-48c0-b59e-68e149119bed","Type":"ContainerDied","Data":"2763f20a91b59e8b3b41928bc95a38ce6ed877f5a7423253ad208cb33df0116a"}
Nov 02 20:04:08.722119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:08.722044    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2763f20a91b59e8b3b41928bc95a38ce6ed877f5a7423253ad208cb33df0116a"
Nov 02 20:04:12.162395 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:12.162350    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:17.071538 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:17.071498    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:21.050761 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:21.050728    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:21.219462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:21.219415    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:21.390469 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:21.390370    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:22.193038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:22.193003    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:22.833519 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:22.833486    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:04:25.081618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:25.081583    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:27.085690 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:27.085652    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:27.362961 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:04:27.362873    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:04:29.685875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:29.685835    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:32.163859 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:32.163820    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:37.191757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:37.191717    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:42.243483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:42.243447    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:47.081019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:47.080986    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:50.084686 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:50.084651    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:50.241736 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:50.241701    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:04:50.781167 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:50.781129    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:51.915415 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:51.915378    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:54.391324 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:54.391280    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:56.808679 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:04:56.808645    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:04:57.360320 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:04:57.360281    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:05:01.931251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:01.931217    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:02.056018 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.055984    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"]
Nov 02 20:05:02.056405 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.056391    5224 cpu_manager.go:414] "containerMap: removing container" podUID="deceea06-2be9-48c0-b59e-68e149119bed" containerName="revoke-keys"
Nov 02 20:05:02.056449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.056407    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deceea06-2be9-48c0-b59e-68e149119bed" containerName="revoke-keys"
Nov 02 20:05:02.056493 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.056484    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="deceea06-2be9-48c0-b59e-68e149119bed" containerName="revoke-keys"
Nov 02 20:05:02.059416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.059400    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.112705 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.112678    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-426941-openshift-default-dockercfg-mxlvf\""
Nov 02 20:05:02.137267 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.137238    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.137378 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.137275    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.137378 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.137294    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.137450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.137397    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.137450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.137424    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.137450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.137443    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.137600 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.137585    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.137639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.137621    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd8n6\" (UniqueName: \"kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-kube-api-access-bd8n6\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.137705 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.137693    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.165968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.165928    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"]
Nov 02 20:05:02.238609 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.238513    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd8n6\" (UniqueName: \"kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-kube-api-access-bd8n6\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.238609 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.238577    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.238893 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.238625    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.238893 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.238662    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.238893 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.238793    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.239065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.238891    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.239065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.238922    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.239065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.238992    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.239224 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.239065    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.239224 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.239096    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.239348 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.239326    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.239390 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.239355    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.239449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.239430    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.239491 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.239466    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.241188 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.241157    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.241188 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.241173    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.296928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.296890    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.297106 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.297082    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd8n6\" (UniqueName: \"kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-kube-api-access-bd8n6\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.371437 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.371399    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:02.784684 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:05:02.784652    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f254db_a4e4_4fed_ac7d_d8b61600091e.slice/crio-47d62c5ecf9cca11216f8ac234bf93bac6304fc055a1566f1dc25a21d5c6bf66 WatchSource:0}: Error finding container 47d62c5ecf9cca11216f8ac234bf93bac6304fc055a1566f1dc25a21d5c6bf66: Status 404 returned error can't find the container with id 47d62c5ecf9cca11216f8ac234bf93bac6304fc055a1566f1dc25a21d5c6bf66
Nov 02 20:05:02.786927 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.786892    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:05:02.787037 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.786960    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:05:02.787037 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.786988    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:05:02.787464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:02.787374    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"]
Nov 02 20:05:03.028403 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:03.028369    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" event={"ID":"d3f254db-a4e4-4fed-ac7d-d8b61600091e","Type":"ContainerStarted","Data":"3b5c5d1b38bfcf72be4b7868a46250b5a1f37ec91a1578c4904326ddcc29a4f1"}
Nov 02 20:05:03.028896 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:03.028421    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" event={"ID":"d3f254db-a4e4-4fed-ac7d-d8b61600091e","Type":"ContainerStarted","Data":"47d62c5ecf9cca11216f8ac234bf93bac6304fc055a1566f1dc25a21d5c6bf66"}
Nov 02 20:05:03.147660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:03.147548    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" podStartSLOduration=1.147529499 podStartE2EDuration="1.147529499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:03.146625376 +0000 UTC m=+984.144313970" watchObservedRunningTime="2026-11-02 20:05:03.147529499 +0000 UTC m=+984.145218110"
Nov 02 20:05:03.374048 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:03.373998    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:03.375520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:03.375495    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 20:05:03.375596 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:03.375552    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" podUID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 20:05:04.372179 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:04.372141    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 20:05:04.372864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:04.372827    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" podUID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 20:05:05.374174 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:05.374127    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 20:05:05.374655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:05.374188    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" podUID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 20:05:06.373191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:06.373151    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 20:05:06.373473 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:06.373442    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" podUID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 20:05:06.975348 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:06.975310    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:07.374261 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:07.374216    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 20:05:07.374441 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:07.374289    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" podUID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 20:05:08.374225 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:08.374184    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 20:05:08.374730 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:08.374251    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" podUID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 20:05:09.419399 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:09.419365    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:09.419918 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:09.419755    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:09.420529 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:09.420510    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 20:05:09.984624 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:09.984590    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"]
Nov 02 20:05:09.988432 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:09.988410    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.020789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.020736    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.021019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.020859    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.021019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.020893    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.021019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.020991    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.021215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.021022    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.021215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.021137    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmgg\" (UniqueName: \"kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-kube-api-access-tbmgg\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.021215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.021184    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.021215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.021210    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.021388 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.021368    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.043661 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.043625    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"]
Nov 02 20:05:10.121915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.121878    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmgg\" (UniqueName: \"kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-kube-api-access-tbmgg\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.122072 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.121930    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.122072 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.121948    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.122072 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.121997    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.122072 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.122046    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.122316 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.122105    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.122316 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.122157    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.122316 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.122206    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.122316 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.122244    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.122519 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.122449    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.122838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.122810    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.122977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.122868    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.122977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.122879    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.123113 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.123090    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.124292 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.124271    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.124491 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.124474    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.180086 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.180051    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.180271 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.180124    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmgg\" (UniqueName: \"kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-kube-api-access-tbmgg\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kdpgz\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.301562 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.301524    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:10.724923 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:05:10.724886    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecfc55c_fb22_4c8a_a63d_ecd9917c2a53.slice/crio-f992672dba8e39371b05b32b7aa6ec2c384370ecd5552e8209ae9e3ce7a6d6f4 WatchSource:0}: Error finding container f992672dba8e39371b05b32b7aa6ec2c384370ecd5552e8209ae9e3ce7a6d6f4: Status 404 returned error can't find the container with id f992672dba8e39371b05b32b7aa6ec2c384370ecd5552e8209ae9e3ce7a6d6f4
Nov 02 20:05:10.727399 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.727358    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:05:10.727472 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.727444    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:05:10.727512 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.727485    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:05:10.728180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:10.728162    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"]
Nov 02 20:05:11.096785 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:11.096733    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" event={"ID":"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53","Type":"ContainerStarted","Data":"fad2b33fbce9ece61a3c94560ca483606ff1c3c9071f84328084e64357ed72fd"}
Nov 02 20:05:11.097056 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:11.096859    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" event={"ID":"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53","Type":"ContainerStarted","Data":"f992672dba8e39371b05b32b7aa6ec2c384370ecd5552e8209ae9e3ce7a6d6f4"}
Nov 02 20:05:11.302058 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:11.302023    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:11.303737 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:11.303712    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 20:05:11.303847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:11.303798    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" podUID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 20:05:11.797881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:11.797839    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:12.302747 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:12.302712    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 20:05:12.302964 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:12.302789    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" podUID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 20:05:13.295436 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.295386    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" podStartSLOduration=4.295371642 podStartE2EDuration="4.295371642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:11.241358505 +0000 UTC m=+992.239047190" watchObservedRunningTime="2026-11-02 20:05:13.295371642 +0000 UTC m=+994.293060298"
Nov 02 20:05:13.295851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.295527    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"]
Nov 02 20:05:13.299078 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.299062    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.302417 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.302396    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 20:05:13.302529 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.302450    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" podUID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 20:05:13.376375 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.376330    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-dshm\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.376567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.376394    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tmp-dir\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.376567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.376493    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-home\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.376567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.376525    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-model-cache\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.376754 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.376553    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wm6r\" (UniqueName: \"kubernetes.io/projected/194602b6-7842-4e1a-bd3c-6bfa00217ca7-kube-api-access-4wm6r\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.376754 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.376624    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tls-certs\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.379527 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.379498    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"]
Nov 02 20:05:13.400431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.400407    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"e2e-conflict-a-58923238-kserve-self-signed-certs\""
Nov 02 20:05:13.436595 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.436565    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:13.477608 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.477561    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tls-certs\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.477837 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.477667    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-dshm\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.477837 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.477701    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tmp-dir\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.477837 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.477804    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-home\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.478014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.477836    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-model-cache\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.478014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.477863    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wm6r\" (UniqueName: \"kubernetes.io/projected/194602b6-7842-4e1a-bd3c-6bfa00217ca7-kube-api-access-4wm6r\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.478572 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.478477    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-model-cache\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.478572 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.478485    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-home\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.479389 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.479366    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tmp-dir\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.480974 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.480934    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-dshm\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.481432 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.481407    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tls-certs\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.536075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.536037    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wm6r\" (UniqueName: \"kubernetes.io/projected/194602b6-7842-4e1a-bd3c-6bfa00217ca7-kube-api-access-4wm6r\") pod \"e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:13.611499 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:13.611410    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:14.041743 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:05:14.041707    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod194602b6_7842_4e1a_bd3c_6bfa00217ca7.slice/crio-60955f582a7c3790dcea50248a69c8df4f0e976cd6590ad359ea3c4ad1d78c5d WatchSource:0}: Error finding container 60955f582a7c3790dcea50248a69c8df4f0e976cd6590ad359ea3c4ad1d78c5d: Status 404 returned error can't find the container with id 60955f582a7c3790dcea50248a69c8df4f0e976cd6590ad359ea3c4ad1d78c5d
Nov 02 20:05:14.045630 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:14.045601    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"]
Nov 02 20:05:14.113581 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:14.113552    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt" event={"ID":"194602b6-7842-4e1a-bd3c-6bfa00217ca7","Type":"ContainerStarted","Data":"60955f582a7c3790dcea50248a69c8df4f0e976cd6590ad359ea3c4ad1d78c5d"}
Nov 02 20:05:14.302822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:14.302678    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 20:05:14.302822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:14.302743    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" podUID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 20:05:15.120221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:15.120186    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt" event={"ID":"194602b6-7842-4e1a-bd3c-6bfa00217ca7","Type":"ContainerStarted","Data":"8e999fdb4108e706ecaf8c5aff75cc062ccff00f7d506d1d9ca87a664ea82c1f"}
Nov 02 20:05:15.236202 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:15.236149    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt" podStartSLOduration=2.23613272 podStartE2EDuration="2.23613272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:15.235521758 +0000 UTC m=+996.233210366" watchObservedRunningTime="2026-11-02 20:05:15.23613272 +0000 UTC m=+996.233821326"
Nov 02 20:05:15.302228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:15.302186    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 20:05:15.302423 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:15.302243    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" podUID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 20:05:15.611695 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:15.611653    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:16.302511 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:16.302468    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 20:05:16.302706 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:16.302533    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" podUID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 20:05:16.790459 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:16.790415    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:17.302070 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:17.302029    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 20:05:17.302253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:17.302103    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" podUID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 20:05:18.766442 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.766359    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"]
Nov 02 20:05:18.770090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.770068    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.823524 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.823484    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c25349-openshift-default-dockercfg-5wzgl\""
Nov 02 20:05:18.834242 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.834201    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fff547-0622-43a8-8305-300fedb8a311-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.834412 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.834262    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fff547-0622-43a8-8305-300fedb8a311-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.834412 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.834382    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.834499 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.834426    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.834499 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.834446    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.834618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.834497    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.834618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.834586    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.834705 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.834638    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7gz6\" (UniqueName: \"kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-kube-api-access-m7gz6\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.834802 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.834749    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.878036 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.877997    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"]
Nov 02 20:05:18.936217 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.936178    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7gz6\" (UniqueName: \"kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-kube-api-access-m7gz6\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.936397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.936247    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.936397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.936379    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fff547-0622-43a8-8305-300fedb8a311-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.936512 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.936415    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fff547-0622-43a8-8305-300fedb8a311-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.936512 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.936456    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.936617 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.936510    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.936617 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.936544    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.936760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.936712    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.936843 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.936795    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.936896 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.936846    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.937173 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.937124    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.937475 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.937450    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fff547-0622-43a8-8305-300fedb8a311-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.937733 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.937699    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.938024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.938002    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.939000 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.938967    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.939745 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.939721    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fff547-0622-43a8-8305-300fedb8a311-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.995114 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.995075    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7gz6\" (UniqueName: \"kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-kube-api-access-m7gz6\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:18.995308 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:18.995075    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:19.084640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:19.084548    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:19.125597 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:19.125563    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:19.126222 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:19.126171    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:19.127237 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:19.127208    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:05:19.467523 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:05:19.467492    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fff547_0622_43a8_8305_300fedb8a311.slice/crio-ec2e2310a987ae3cc9ab9335f786a2f920c5e398ec2dcb66688dc2d42c8cf629 WatchSource:0}: Error finding container ec2e2310a987ae3cc9ab9335f786a2f920c5e398ec2dcb66688dc2d42c8cf629: Status 404 returned error can't find the container with id ec2e2310a987ae3cc9ab9335f786a2f920c5e398ec2dcb66688dc2d42c8cf629
Nov 02 20:05:19.469978 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:19.469947    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:05:19.470068 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:19.470014    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:05:19.470068 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:19.470049    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:05:19.470517 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:19.470497    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"]
Nov 02 20:05:19.903174 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:19.903135    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:05:20.180291 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:20.180197    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w" event={"ID":"b0fff547-0622-43a8-8305-300fedb8a311","Type":"ContainerStarted","Data":"3b19830a004e899b4ca79b7ec3ac8b5fb7b28282980fc94974fe49fb7e5921d0"}
Nov 02 20:05:20.180291 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:20.180264    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w" event={"ID":"b0fff547-0622-43a8-8305-300fedb8a311","Type":"ContainerStarted","Data":"ec2e2310a987ae3cc9ab9335f786a2f920c5e398ec2dcb66688dc2d42c8cf629"}
Nov 02 20:05:20.299179 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:20.299123    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w" podStartSLOduration=2.299102359 podStartE2EDuration="2.299102359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:20.298195358 +0000 UTC m=+1001.295883963" watchObservedRunningTime="2026-11-02 20:05:20.299102359 +0000 UTC m=+1001.296790967"
Nov 02 20:05:21.085717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:21.085679    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:21.086953 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:21.086922    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 20:05:21.087081 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:21.086971    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w" podUID="b0fff547-0622-43a8-8305-300fedb8a311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 20:05:21.786940 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:21.786866    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:22.085759 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:22.085675    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 20:05:22.085759 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:22.085729    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w" podUID="b0fff547-0622-43a8-8305-300fedb8a311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 20:05:23.085055 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:23.085018    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 20:05:23.085281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:23.085081    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w" podUID="b0fff547-0622-43a8-8305-300fedb8a311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 20:05:23.612019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:23.611977    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:23.624618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:23.624582    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:24.085865 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:24.085828    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 20:05:24.086074 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:24.085885    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w" podUID="b0fff547-0622-43a8-8305-300fedb8a311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 20:05:24.213800 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:24.213749    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:05:25.085877 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:25.085840    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 20:05:25.086300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:25.085899    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w" podUID="b0fff547-0622-43a8-8305-300fedb8a311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 20:05:26.221590 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:26.221554    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:26.222516 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:26.222494    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:26.223377 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:26.223355    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:05:26.720797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:26.720731    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:27.357908 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:05:27.357872    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:05:31.571676 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:31.571631    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:36.559726 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:36.559689    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:41.456619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:41.456574    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:46.705954 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:46.705904    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:05:46.706362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:46.705976    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" podUID="a9bd828b-4265-447e-89da-c94e97503b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:05:46.932783 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:46.932732    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:50.435385 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:50.435343    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:51.431393 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:51.431355    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:51.755409 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:51.755310    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:53.830523 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:53.830492    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:56.780033 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:05:56.779986    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:05:57.359723 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:05:57.359678    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:06:01.721490 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:01.721452    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:06:01.761162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:01.761117    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:02.070112 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:02.070079    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:02.246304 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:02.246264    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:02.425837 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:02.425089    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:05.342280 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:05.342232    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:06.779372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:06.779285    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:11.755892 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:11.755854    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:16.987177 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:16.987134    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:21.878817 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:21.878740    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:26.853299 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:26.853264    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:27.358957 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:06:27.358920    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:06:28.737532 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:28.737484    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:06:30.574832 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.574788    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"]
Nov 02 20:06:30.590206 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.590171    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.631492 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.631447    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"]
Nov 02 20:06:30.723585 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.723550    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"e2e-conflict-b-58923238-kserve-self-signed-certs\""
Nov 02 20:06:30.729915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.724985    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:30.746899 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.746865    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-dshm\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.747085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.746924    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tmp-dir\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.747085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.747031    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tls-certs\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.747085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.747068    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-home\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.747222 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.747089    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-model-cache\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.747222 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.747118    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xg6m\" (UniqueName: \"kubernetes.io/projected/a6c9e93e-28e7-4145-b5ad-015c1a55753c-kube-api-access-5xg6m\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.848989 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.848881    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xg6m\" (UniqueName: \"kubernetes.io/projected/a6c9e93e-28e7-4145-b5ad-015c1a55753c-kube-api-access-5xg6m\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.849352 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.849325    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-dshm\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.849546 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.849526    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tmp-dir\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.849706 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.849690    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tls-certs\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.853934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.853906    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-home\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.854125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.854107    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-model-cache\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.854616 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.854592    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-model-cache\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.855063 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.855039    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-home\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.855214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.854022    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tmp-dir\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.864105 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.864061    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-dshm\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.905805 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.902893    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tls-certs\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.923841 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.914801    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xg6m\" (UniqueName: \"kubernetes.io/projected/a6c9e93e-28e7-4145-b5ad-015c1a55753c-kube-api-access-5xg6m\") pod \"e2e-conflict-b-58923238-kserve-56b4887845-pngvq\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") " pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:30.943038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:30.942994    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:31.707691 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:06:31.707654    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6c9e93e_28e7_4145_b5ad_015c1a55753c.slice/crio-c88ba40956c802c7d500186ffc33ec092f9d148a1430b81751e907d85df6d742 WatchSource:0}: Error finding container c88ba40956c802c7d500186ffc33ec092f9d148a1430b81751e907d85df6d742: Status 404 returned error can't find the container with id c88ba40956c802c7d500186ffc33ec092f9d148a1430b81751e907d85df6d742
Nov 02 20:06:31.714023 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:31.711867    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:06:31.714210 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:31.714181    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" podUID="a9bd828b-4265-447e-89da-c94e97503b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:06:31.714725 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:31.708714    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"]
Nov 02 20:06:32.075111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:32.074895    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:06:32.075111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:32.074976    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" podUID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:06:32.350273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:32.350184    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:32.638390 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:32.638294    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq" event={"ID":"a6c9e93e-28e7-4145-b5ad-015c1a55753c","Type":"ContainerStarted","Data":"258be80e8e8ea272e346faf972c4ec8496c79c2b6f56205565f628811bc0005f"}
Nov 02 20:06:32.638390 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:32.638339    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq" event={"ID":"a6c9e93e-28e7-4145-b5ad-015c1a55753c","Type":"ContainerStarted","Data":"c88ba40956c802c7d500186ffc33ec092f9d148a1430b81751e907d85df6d742"}
Nov 02 20:06:32.755412 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:32.755360    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq" podStartSLOduration=2.755345712 podStartE2EDuration="2.755345712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:32.754054445 +0000 UTC m=+1073.751743053" watchObservedRunningTime="2026-11-02 20:06:32.755345712 +0000 UTC m=+1073.753034321"
Nov 02 20:06:32.944014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:32.943927    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:32.972550 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:32.972508    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:37.282749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:37.282694    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:39.298399 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:39.298363    5224 scope.go:122] "RemoveContainer" containerID="61de71598422feaa6b40a43adbdafb470ab1454b8fe2b6f3b8c2f59647d889a6"
Nov 02 20:06:39.307487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:39.307464    5224 scope.go:122] "RemoveContainer" containerID="c8f3e01b3e09305b99247704d0140505e7bb3fce4cbf102a94dc69eb46af6e50"
Nov 02 20:06:39.315819 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:39.315794    5224 scope.go:122] "RemoveContainer" containerID="0a3d81ea410fb1c0131db9cdcac74bbdaf18ad09778948c01a708be9d02b4199"
Nov 02 20:06:39.324066 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:39.324044    5224 scope.go:122] "RemoveContainer" containerID="3cfe4f2dd1f1aa60f34ea1c3516d5ed200b274e04c146890ba43873fbfa83bae"
Nov 02 20:06:40.944308 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:40.944253    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:40.957148 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:40.957119    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:41.700946 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:41.700914    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:06:42.245715 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:42.245671    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:47.058836 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:47.058795    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:50.253505 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:50.253424    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:06:50.754515 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:50.754476    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:51.464233 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:51.464197    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:51.672582 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:51.672541    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:51.953534 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:51.953492    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:56.873662 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:06:56.873626    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:06:57.357931 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:06:57.357899    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:07:01.766675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:01.766632    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:06.657265 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:06.657224    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:11.561952 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:11.561907    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:12.448997 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:12.448964    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:07:14.982883 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:14.982847    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"]
Nov 02 20:07:14.987055 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:14.987031    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.038561 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.038521    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"]
Nov 02 20:07:15.086441 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.086411    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c25349\"/\"distinct-2-w4-c25349-kserve-self-signed-certs\""
Nov 02 20:07:15.146722 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.146687    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:15.161093 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.161050    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3359dc52-961f-43fd-b567-3bcb22271302-tls-certs\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.161268 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.161172    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-home\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.161268 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.161215    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-model-cache\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.161366 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.161344    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-dshm\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.161423 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.161407    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwm8b\" (UniqueName: \"kubernetes.io/projected/3359dc52-961f-43fd-b567-3bcb22271302-kube-api-access-rwm8b\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.161464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.161444    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-tmp-dir\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.262405 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.262320    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3359dc52-961f-43fd-b567-3bcb22271302-tls-certs\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.262578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.262435    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-home\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.262578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.262473    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-model-cache\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.262700 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.262586    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-dshm\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.262700 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.262664    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwm8b\" (UniqueName: \"kubernetes.io/projected/3359dc52-961f-43fd-b567-3bcb22271302-kube-api-access-rwm8b\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.262840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.262717    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-tmp-dir\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.263131 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.263101    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-tmp-dir\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.263244 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.263167    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-model-cache\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.263539 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.263514    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-home\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.265227 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.265203    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-dshm\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.265660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.265631    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3359dc52-961f-43fd-b567-3bcb22271302-tls-certs\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.320993 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.320950    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwm8b\" (UniqueName: \"kubernetes.io/projected/3359dc52-961f-43fd-b567-3bcb22271302-kube-api-access-rwm8b\") pod \"distinct-2-w4-c25349-kserve-548f4b7884-qkszh\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.600925 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.600884    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:15.964243 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:07:15.964215    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3359dc52_961f_43fd_b567_3bcb22271302.slice/crio-e62c2f6ec156d24d086d5090c51b2fbd95872b3a00a290c339a7f424cd2cf125 WatchSource:0}: Error finding container e62c2f6ec156d24d086d5090c51b2fbd95872b3a00a290c339a7f424cd2cf125: Status 404 returned error can't find the container with id e62c2f6ec156d24d086d5090c51b2fbd95872b3a00a290c339a7f424cd2cf125
Nov 02 20:07:15.967453 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:15.967429    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"]
Nov 02 20:07:16.481881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:16.481845    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:16.883709 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:16.883671    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh" event={"ID":"3359dc52-961f-43fd-b567-3bcb22271302","Type":"ContainerStarted","Data":"6c35ac8eda2718c2774bd2ef412b1abc6410f3c821a53ebfe6bf8ae0cb5abc23"}
Nov 02 20:07:16.883935 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:16.883732    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh" event={"ID":"3359dc52-961f-43fd-b567-3bcb22271302","Type":"ContainerStarted","Data":"e62c2f6ec156d24d086d5090c51b2fbd95872b3a00a290c339a7f424cd2cf125"}
Nov 02 20:07:17.000742 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:17.000694    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh" podStartSLOduration=3.000680693 podStartE2EDuration="3.000680693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:16.999741765 +0000 UTC m=+1117.997430365" watchObservedRunningTime="2026-11-02 20:07:17.000680693 +0000 UTC m=+1117.998369303"
Nov 02 20:07:17.601926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:17.601892    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:21.363228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:21.363194    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:25.601351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:25.601313    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:25.614100 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:25.614077    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:25.943018 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:25.942934    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:07:26.266349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:26.266287    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:27.357644 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:07:27.357614    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:07:31.141864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:31.141823    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:36.055953 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:36.055918    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:40.976177 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:40.976142    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:41.351376 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:41.351340    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:07:45.021600 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.021568    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"]
Nov 02 20:07:45.024739 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.024721    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.077991 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.077963    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"]
Nov 02 20:07:45.093998 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.093966    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:45.127036 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.127011    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-14d090\"/\"distinct-2-w0-14d090-kserve-self-signed-certs\""
Nov 02 20:07:45.181462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.181428    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-home\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.181462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.181464    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9l8w\" (UniqueName: \"kubernetes.io/projected/c50013f5-4bb1-4b40-b62f-5b736b953ac0-kube-api-access-v9l8w\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.181686 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.181569    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tls-certs\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.181686 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.181654    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tmp-dir\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.181800 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.181704    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-model-cache\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.181800 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.181748    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-dshm\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.283245 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.283161    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-home\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.283245 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.283196    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9l8w\" (UniqueName: \"kubernetes.io/projected/c50013f5-4bb1-4b40-b62f-5b736b953ac0-kube-api-access-v9l8w\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.283245 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.283231    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tls-certs\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.283531 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.283264    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tmp-dir\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.283531 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.283314    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-model-cache\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.283531 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.283348    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-dshm\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.283685 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.283625    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-home\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.283744 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.283697    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tmp-dir\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.283868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.283845    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-model-cache\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.285564 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.285545    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-dshm\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.285875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.285855    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tls-certs\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.341163 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.341127    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9l8w\" (UniqueName: \"kubernetes.io/projected/c50013f5-4bb1-4b40-b62f-5b736b953ac0-kube-api-access-v9l8w\") pod \"distinct-2-w0-14d090-kserve-6b58bcf874-n22gs\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.636472 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.636379    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:45.892708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:45.891894    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:46.004234 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:07:46.004195    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc50013f5_4bb1_4b40_b62f_5b736b953ac0.slice/crio-36697a7b3f560d98da048b70e0aca8ebdc113f4b069d7076e2ace949b6dc20c9 WatchSource:0}: Error finding container 36697a7b3f560d98da048b70e0aca8ebdc113f4b069d7076e2ace949b6dc20c9: Status 404 returned error can't find the container with id 36697a7b3f560d98da048b70e0aca8ebdc113f4b069d7076e2ace949b6dc20c9
Nov 02 20:07:46.007471 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:46.007449    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"]
Nov 02 20:07:46.043457 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:46.043424    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs" event={"ID":"c50013f5-4bb1-4b40-b62f-5b736b953ac0","Type":"ContainerStarted","Data":"36697a7b3f560d98da048b70e0aca8ebdc113f4b069d7076e2ace949b6dc20c9"}
Nov 02 20:07:47.051845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:47.051808    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs" event={"ID":"c50013f5-4bb1-4b40-b62f-5b736b953ac0","Type":"ContainerStarted","Data":"5d8d935c4ecc65a97b146839b952f08d74110cb2d836c33358326a5c0db685b1"}
Nov 02 20:07:47.169458 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:47.169404    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs" podStartSLOduration=3.169392072 podStartE2EDuration="3.169392072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:47.167576599 +0000 UTC m=+1148.165265207" watchObservedRunningTime="2026-11-02 20:07:47.169392072 +0000 UTC m=+1148.167080734"
Nov 02 20:07:47.232953 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:47.232920    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"]
Nov 02 20:07:47.317807 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:47.317701    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:47.636742 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:47.636672    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:50.758186 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:50.758148    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:51.089120 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:51.089087    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:51.810415 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:51.810379    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:55.637241 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:55.637189    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:55.650625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:55.650601    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:55.724281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:55.724244    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:07:56.121743 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:56.121710    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:07:57.358139 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:07:57.358105    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:07:58.482609 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:07:58.482575    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:00.524966 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:00.524935    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"]
Nov 02 20:08:00.587060 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:00.587029    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:00.765696 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:00.765662    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:02.255679 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.255638    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq" podUID="a6c9e93e-28e7-4145-b5ad-015c1a55753c" containerName="main" containerID="cri-o://258be80e8e8ea272e346faf972c4ec8496c79c2b6f56205565f628811bc0005f" gracePeriod=45
Nov 02 20:08:02.538272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.538243    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:08:02.586122 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.586093    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-model-cache\") pod \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") "
Nov 02 20:08:02.586278 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.586134    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tls-certs\") pod \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") "
Nov 02 20:08:02.586278 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.586187    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-home\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-home\") pod \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") "
Nov 02 20:08:02.586393 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.586318    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tmp-dir\") pod \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") "
Nov 02 20:08:02.586393 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.586354    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c9e93e-28e7-4145-b5ad-015c1a55753c-kube-api-access-5xg6m\" (UniqueName: \"kubernetes.io/projected/a6c9e93e-28e7-4145-b5ad-015c1a55753c-kube-api-access-5xg6m\") pod \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") "
Nov 02 20:08:02.586486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.586384    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-model-cache" pod "a6c9e93e-28e7-4145-b5ad-015c1a55753c" (UID: "a6c9e93e-28e7-4145-b5ad-015c1a55753c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:02.586486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.586407    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-dshm\") pod \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\" (UID: \"a6c9e93e-28e7-4145-b5ad-015c1a55753c\") "
Nov 02 20:08:02.586572 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.586542    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tmp-dir" pod "a6c9e93e-28e7-4145-b5ad-015c1a55753c" (UID: "a6c9e93e-28e7-4145-b5ad-015c1a55753c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:02.586572 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.586554    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-home" pod "a6c9e93e-28e7-4145-b5ad-015c1a55753c" (UID: "a6c9e93e-28e7-4145-b5ad-015c1a55753c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:02.586790 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.586754    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:02.586896 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.586805    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:02.586896 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.586818    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:02.588446 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.588423    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c9e93e-28e7-4145-b5ad-015c1a55753c-kube-api-access-5xg6m" pod "a6c9e93e-28e7-4145-b5ad-015c1a55753c" (UID: "a6c9e93e-28e7-4145-b5ad-015c1a55753c"). InnerVolumeSpecName "kube-api-access-5xg6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:02.588542 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.588445    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tls-certs" pod "a6c9e93e-28e7-4145-b5ad-015c1a55753c" (UID: "a6c9e93e-28e7-4145-b5ad-015c1a55753c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:08:02.588542 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.588503    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-dshm" pod "a6c9e93e-28e7-4145-b5ad-015c1a55753c" (UID: "a6c9e93e-28e7-4145-b5ad-015c1a55753c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:02.687589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.687560    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xg6m\" (UniqueName: \"kubernetes.io/projected/a6c9e93e-28e7-4145-b5ad-015c1a55753c-kube-api-access-5xg6m\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:02.687589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.687589    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c9e93e-28e7-4145-b5ad-015c1a55753c-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:02.687795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:02.687605    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c9e93e-28e7-4145-b5ad-015c1a55753c-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:03.164484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:03.164449    5224 generic.go:356] "Generic (PLEG): container finished" podID="a6c9e93e-28e7-4145-b5ad-015c1a55753c" containerID="258be80e8e8ea272e346faf972c4ec8496c79c2b6f56205565f628811bc0005f" exitCode=0
Nov 02 20:08:03.164674 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:03.164562    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"
Nov 02 20:08:03.166584 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:03.166560    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq" event={"ID":"a6c9e93e-28e7-4145-b5ad-015c1a55753c","Type":"ContainerDied","Data":"258be80e8e8ea272e346faf972c4ec8496c79c2b6f56205565f628811bc0005f"}
Nov 02 20:08:03.166717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:03.166592    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq" event={"ID":"a6c9e93e-28e7-4145-b5ad-015c1a55753c","Type":"ContainerDied","Data":"c88ba40956c802c7d500186ffc33ec092f9d148a1430b81751e907d85df6d742"}
Nov 02 20:08:03.166717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:03.166609    5224 scope.go:122] "RemoveContainer" containerID="258be80e8e8ea272e346faf972c4ec8496c79c2b6f56205565f628811bc0005f"
Nov 02 20:08:03.176819 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:03.176804    5224 scope.go:122] "RemoveContainer" containerID="258be80e8e8ea272e346faf972c4ec8496c79c2b6f56205565f628811bc0005f"
Nov 02 20:08:03.177051 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:08:03.177032    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258be80e8e8ea272e346faf972c4ec8496c79c2b6f56205565f628811bc0005f\": container with ID starting with 258be80e8e8ea272e346faf972c4ec8496c79c2b6f56205565f628811bc0005f not found: ID does not exist" containerID="258be80e8e8ea272e346faf972c4ec8496c79c2b6f56205565f628811bc0005f"
Nov 02 20:08:03.177113 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:03.177059    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258be80e8e8ea272e346faf972c4ec8496c79c2b6f56205565f628811bc0005f"} err="failed to get container status \"258be80e8e8ea272e346faf972c4ec8496c79c2b6f56205565f628811bc0005f\": rpc error: code = NotFound desc = could not find container \"258be80e8e8ea272e346faf972c4ec8496c79c2b6f56205565f628811bc0005f\": container with ID starting with 258be80e8e8ea272e346faf972c4ec8496c79c2b6f56205565f628811bc0005f not found: ID does not exist"
Nov 02 20:08:03.331565 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:03.331532    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"]
Nov 02 20:08:03.382822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:03.382791    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/e2e-conflict-b-58923238-kserve-56b4887845-pngvq"]
Nov 02 20:08:05.166938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:05.166900    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c9e93e-28e7-4145-b5ad-015c1a55753c" path="/var/lib/kubelet/pods/a6c9e93e-28e7-4145-b5ad-015c1a55753c/volumes"
Nov 02 20:08:05.676954 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:05.676913    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:10.553236 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:10.553202    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:14.206276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:14.206244    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:08:15.507323 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:15.507285    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:15.571406 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:15.571365    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt" podUID="194602b6-7842-4e1a-bd3c-6bfa00217ca7" containerName="main" containerID="cri-o://8e999fdb4108e706ecaf8c5aff75cc062ccff00f7d506d1d9ca87a664ea82c1f" gracePeriod=45
Nov 02 20:08:15.880663 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:15.880630    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:08:16.030108 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.030075    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tls-certs\" (UniqueName: \"kubernetes.io/secret/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tls-certs\") pod \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") "
Nov 02 20:08:16.030282 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.030119    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-model-cache\") pod \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") "
Nov 02 20:08:16.030282 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.030192    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-dshm\") pod \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") "
Nov 02 20:08:16.030282 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.030252    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tmp-dir\") pod \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") "
Nov 02 20:08:16.030423 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.030338    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194602b6-7842-4e1a-bd3c-6bfa00217ca7-kube-api-access-4wm6r\" (UniqueName: \"kubernetes.io/projected/194602b6-7842-4e1a-bd3c-6bfa00217ca7-kube-api-access-4wm6r\") pod \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") "
Nov 02 20:08:16.030423 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.030371    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-home\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-home\") pod \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\" (UID: \"194602b6-7842-4e1a-bd3c-6bfa00217ca7\") "
Nov 02 20:08:16.030423 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.030385    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-model-cache" pod "194602b6-7842-4e1a-bd3c-6bfa00217ca7" (UID: "194602b6-7842-4e1a-bd3c-6bfa00217ca7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:16.030603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.030549    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tmp-dir" pod "194602b6-7842-4e1a-bd3c-6bfa00217ca7" (UID: "194602b6-7842-4e1a-bd3c-6bfa00217ca7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:16.030669 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.030620    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-home" pod "194602b6-7842-4e1a-bd3c-6bfa00217ca7" (UID: "194602b6-7842-4e1a-bd3c-6bfa00217ca7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:16.030793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.030758    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:16.030841 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.030800    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:16.030841 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.030814    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:16.032428 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.032400    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-dshm" pod "194602b6-7842-4e1a-bd3c-6bfa00217ca7" (UID: "194602b6-7842-4e1a-bd3c-6bfa00217ca7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:16.032546 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.032437    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194602b6-7842-4e1a-bd3c-6bfa00217ca7-kube-api-access-4wm6r" pod "194602b6-7842-4e1a-bd3c-6bfa00217ca7" (UID: "194602b6-7842-4e1a-bd3c-6bfa00217ca7"). InnerVolumeSpecName "kube-api-access-4wm6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:16.032546 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.032484    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tls-certs" pod "194602b6-7842-4e1a-bd3c-6bfa00217ca7" (UID: "194602b6-7842-4e1a-bd3c-6bfa00217ca7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:08:16.131879 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.131846    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194602b6-7842-4e1a-bd3c-6bfa00217ca7-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:16.131879 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.131879    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/194602b6-7842-4e1a-bd3c-6bfa00217ca7-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:16.132068 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.131893    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wm6r\" (UniqueName: \"kubernetes.io/projected/194602b6-7842-4e1a-bd3c-6bfa00217ca7-kube-api-access-4wm6r\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:16.234403 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.234373    5224 generic.go:356] "Generic (PLEG): container finished" podID="194602b6-7842-4e1a-bd3c-6bfa00217ca7" containerID="8e999fdb4108e706ecaf8c5aff75cc062ccff00f7d506d1d9ca87a664ea82c1f" exitCode=0
Nov 02 20:08:16.234580 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.234414    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"
Nov 02 20:08:16.234580 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.234455    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt" event={"ID":"194602b6-7842-4e1a-bd3c-6bfa00217ca7","Type":"ContainerDied","Data":"8e999fdb4108e706ecaf8c5aff75cc062ccff00f7d506d1d9ca87a664ea82c1f"}
Nov 02 20:08:16.234580 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.234496    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt" event={"ID":"194602b6-7842-4e1a-bd3c-6bfa00217ca7","Type":"ContainerDied","Data":"60955f582a7c3790dcea50248a69c8df4f0e976cd6590ad359ea3c4ad1d78c5d"}
Nov 02 20:08:16.234580 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.234512    5224 scope.go:122] "RemoveContainer" containerID="8e999fdb4108e706ecaf8c5aff75cc062ccff00f7d506d1d9ca87a664ea82c1f"
Nov 02 20:08:16.247465 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.247447    5224 scope.go:122] "RemoveContainer" containerID="8e999fdb4108e706ecaf8c5aff75cc062ccff00f7d506d1d9ca87a664ea82c1f"
Nov 02 20:08:16.247731 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:08:16.247710    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e999fdb4108e706ecaf8c5aff75cc062ccff00f7d506d1d9ca87a664ea82c1f\": container with ID starting with 8e999fdb4108e706ecaf8c5aff75cc062ccff00f7d506d1d9ca87a664ea82c1f not found: ID does not exist" containerID="8e999fdb4108e706ecaf8c5aff75cc062ccff00f7d506d1d9ca87a664ea82c1f"
Nov 02 20:08:16.247806 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.247740    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e999fdb4108e706ecaf8c5aff75cc062ccff00f7d506d1d9ca87a664ea82c1f"} err="failed to get container status \"8e999fdb4108e706ecaf8c5aff75cc062ccff00f7d506d1d9ca87a664ea82c1f\": rpc error: code = NotFound desc = could not find container \"8e999fdb4108e706ecaf8c5aff75cc062ccff00f7d506d1d9ca87a664ea82c1f\": container with ID starting with 8e999fdb4108e706ecaf8c5aff75cc062ccff00f7d506d1d9ca87a664ea82c1f not found: ID does not exist"
Nov 02 20:08:16.401524 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.401441    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"]
Nov 02 20:08:16.406938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:16.406907    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/e2e-conflict-a-58923238-kserve-65b8ddf8ff-nx6lt"]
Nov 02 20:08:17.167194 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:17.167161    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194602b6-7842-4e1a-bd3c-6bfa00217ca7" path="/var/lib/kubelet/pods/194602b6-7842-4e1a-bd3c-6bfa00217ca7/volumes"
Nov 02 20:08:20.466441 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:20.466364    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:25.172215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.172184    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"]
Nov 02 20:08:25.172619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.172589    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a6c9e93e-28e7-4145-b5ad-015c1a55753c" containerName="main"
Nov 02 20:08:25.172619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.172600    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c9e93e-28e7-4145-b5ad-015c1a55753c" containerName="main"
Nov 02 20:08:25.172686 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.172628    5224 cpu_manager.go:414] "containerMap: removing container" podUID="194602b6-7842-4e1a-bd3c-6bfa00217ca7" containerName="main"
Nov 02 20:08:25.172686 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.172634    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194602b6-7842-4e1a-bd3c-6bfa00217ca7" containerName="main"
Nov 02 20:08:25.172749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.172701    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="194602b6-7842-4e1a-bd3c-6bfa00217ca7" containerName="main"
Nov 02 20:08:25.172749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.172711    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c9e93e-28e7-4145-b5ad-015c1a55753c" containerName="main"
Nov 02 20:08:25.175821 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.175804    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.226376 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.226349    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ndkh\" (UniqueName: \"kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-kube-api-access-5ndkh\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.226550 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.226393    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.226550 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.226415    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.226625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.226540    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.226625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.226613    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.226730 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.226670    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.226805 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.226756    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.226860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.226844    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.226896 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.226876    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.229663 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.229646    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-bbee67-openshift-default-dockercfg-lw7rw\""
Nov 02 20:08:25.283266 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.283240    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"]
Nov 02 20:08:25.328268 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.328239    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.328429 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.328290    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.328429 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.328319    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.328429 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.328412    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndkh\" (UniqueName: \"kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-kube-api-access-5ndkh\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.328579 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.328456    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.328579 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.328481    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.328579 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.328506    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.328579 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.328565    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.328779 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.328608    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.328883 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.328855    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.328963 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.328939    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.329036 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.329012    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.329083 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.329058    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.329121 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.329112    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.330499 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.330481    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.331001 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.330983    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.347050 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.347009    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:25.386747 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.386719    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.386954 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.386926    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ndkh\" (UniqueName: \"kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-kube-api-access-5ndkh\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.488828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.488735    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:25.908089 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:08:25.908056    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0d9cbd_fa51_44ea_8333_2b7db2337e89.slice/crio-c9449c2de5c9136960f85a7d05fa8c7c5ac2a74671821b0e2a815a135448807f WatchSource:0}: Error finding container c9449c2de5c9136960f85a7d05fa8c7c5ac2a74671821b0e2a815a135448807f: Status 404 returned error can't find the container with id c9449c2de5c9136960f85a7d05fa8c7c5ac2a74671821b0e2a815a135448807f
Nov 02 20:08:25.910320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.910284    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:08:25.910449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.910374    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:08:25.910449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.910419    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:08:25.911298 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:25.911273    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"]
Nov 02 20:08:26.287698 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:26.287660    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x" event={"ID":"ed0d9cbd-fa51-44ea-8333-2b7db2337e89","Type":"ContainerStarted","Data":"438d750cb13d9a2da07d5ba1de8a784af52522123f05ce288feede5238971ec0"}
Nov 02 20:08:26.288106 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:26.287720    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x" event={"ID":"ed0d9cbd-fa51-44ea-8333-2b7db2337e89","Type":"ContainerStarted","Data":"c9449c2de5c9136960f85a7d05fa8c7c5ac2a74671821b0e2a815a135448807f"}
Nov 02 20:08:26.410632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:26.410580    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x" podStartSLOduration=1.410566264 podStartE2EDuration="1.410566264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:26.408646502 +0000 UTC m=+1187.406335111" watchObservedRunningTime="2026-11-02 20:08:26.410566264 +0000 UTC m=+1187.408254872"
Nov 02 20:08:26.489362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:26.489327    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:26.491186 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:26.491143    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 20:08:26.491304 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:26.491208    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x" podUID="ed0d9cbd-fa51-44ea-8333-2b7db2337e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 20:08:27.385430 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:08:27.385389    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:08:27.489476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:27.489439    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 20:08:27.489657 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:27.489503    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x" podUID="ed0d9cbd-fa51-44ea-8333-2b7db2337e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 20:08:28.489863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:28.489821    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 20:08:28.490249 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:28.489889    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x" podUID="ed0d9cbd-fa51-44ea-8333-2b7db2337e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 20:08:29.490071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:29.490036    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 20:08:29.490468 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:29.490094    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x" podUID="ed0d9cbd-fa51-44ea-8333-2b7db2337e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 20:08:30.279240 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:30.279194    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:30.490097 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:30.490054    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 20:08:30.490508 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:30.490111    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x" podUID="ed0d9cbd-fa51-44ea-8333-2b7db2337e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 20:08:31.489180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:31.489148    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 20:08:31.489362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:31.489203    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x" podUID="ed0d9cbd-fa51-44ea-8333-2b7db2337e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 20:08:32.557936 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:32.557903    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:32.559163 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:32.559129    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:32.559966 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:32.559948    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:08:33.493880 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.493844    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"]
Nov 02 20:08:33.497729 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.497710    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"
Nov 02 20:08:33.538687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.538659    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c727c0-zgh4z\" (UID: \"644fb6e2-8934-4031-a125-8c9fceea7397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"
Nov 02 20:08:33.538836 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.538698    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c727c0-zgh4z\" (UID: \"644fb6e2-8934-4031-a125-8c9fceea7397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"
Nov 02 20:08:33.538836 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.538720    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-465qg\" (UniqueName: \"kubernetes.io/projected/644fb6e2-8934-4031-a125-8c9fceea7397-kube-api-access-465qg\") pod \"maas-api-revoke-sub-33c727c0-zgh4z\" (UID: \"644fb6e2-8934-4031-a125-8c9fceea7397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"
Nov 02 20:08:33.549743 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.549719    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"]
Nov 02 20:08:33.598753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.598723    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:08:33.599133 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.598799    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:08:33.599133 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.598856    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:08:33.601806 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.601783    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:08:33.640303 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.640272    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c727c0-zgh4z\" (UID: \"644fb6e2-8934-4031-a125-8c9fceea7397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"
Nov 02 20:08:33.640461 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.640311    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c727c0-zgh4z\" (UID: \"644fb6e2-8934-4031-a125-8c9fceea7397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"
Nov 02 20:08:33.640461 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.640336    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-465qg\" (UniqueName: \"kubernetes.io/projected/644fb6e2-8934-4031-a125-8c9fceea7397-kube-api-access-465qg\") pod \"maas-api-revoke-sub-33c727c0-zgh4z\" (UID: \"644fb6e2-8934-4031-a125-8c9fceea7397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"
Nov 02 20:08:33.640976 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.640953    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c727c0-zgh4z\" (UID: \"644fb6e2-8934-4031-a125-8c9fceea7397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"
Nov 02 20:08:33.642555 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.642535    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c727c0-zgh4z\" (UID: \"644fb6e2-8934-4031-a125-8c9fceea7397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"
Nov 02 20:08:33.652742 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.652719    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"]
Nov 02 20:08:33.656450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.656432    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.693210 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.693174    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"]
Nov 02 20:08:33.698614 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.698588    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-465qg\" (UniqueName: \"kubernetes.io/projected/644fb6e2-8934-4031-a125-8c9fceea7397-kube-api-access-465qg\") pod \"maas-api-revoke-sub-33c727c0-zgh4z\" (UID: \"644fb6e2-8934-4031-a125-8c9fceea7397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"
Nov 02 20:08:33.741802 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.741745    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbmhc\" (UniqueName: \"kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-kube-api-access-mbmhc\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.741956 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.741848    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.741956 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.741916    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.741956 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.741945    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.742071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.742054    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.742149 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.742132    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.742192 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.742181    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.742226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.742202    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.742301 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.742291    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.810799 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.810749    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"
Nov 02 20:08:33.843361 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.843328    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.843507 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.843379    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.843507 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.843434    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.843507 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.843488    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.843661 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.843516    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.843661 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.843576    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.843661 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.843644    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbmhc\" (UniqueName: \"kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-kube-api-access-mbmhc\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.843863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.843674    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.843863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.843760    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.843863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.843759    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.843863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.843812    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.844174 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.844154    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.844223 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.844202    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.844424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.844408    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.846056 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.846038    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.846385 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.846366    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.901786 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.901732    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbmhc\" (UniqueName: \"kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-kube-api-access-mbmhc\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.902035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.902013    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:33.970024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:33.969991    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:34.175756 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:08:34.175725    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644fb6e2_8934_4031_a125_8c9fceea7397.slice/crio-cf12bb881588a3884cb4ecc6179bde5db311d4edac8d2cf74884d2e0031b2454 WatchSource:0}: Error finding container cf12bb881588a3884cb4ecc6179bde5db311d4edac8d2cf74884d2e0031b2454: Status 404 returned error can't find the container with id cf12bb881588a3884cb4ecc6179bde5db311d4edac8d2cf74884d2e0031b2454
Nov 02 20:08:34.179484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:34.179455    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"]
Nov 02 20:08:34.334350 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:34.334310    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z" event={"ID":"644fb6e2-8934-4031-a125-8c9fceea7397","Type":"ContainerStarted","Data":"cf12bb881588a3884cb4ecc6179bde5db311d4edac8d2cf74884d2e0031b2454"}
Nov 02 20:08:34.337711 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:08:34.337681    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402ae8c8_446f_4812_9ecc_9cb29b9c35fe.slice/crio-c0ace55e88124363ceb094e55819899c02af8611f07ec3e3b50c1cdf22d6c99f WatchSource:0}: Error finding container c0ace55e88124363ceb094e55819899c02af8611f07ec3e3b50c1cdf22d6c99f: Status 404 returned error can't find the container with id c0ace55e88124363ceb094e55819899c02af8611f07ec3e3b50c1cdf22d6c99f
Nov 02 20:08:34.339916 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:34.339884    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:08:34.340019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:34.339949    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:08:34.340019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:34.339979    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:08:34.341091 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:34.341068    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"]
Nov 02 20:08:35.183746 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.183713    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"]
Nov 02 20:08:35.188288 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.188265    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.226116 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.226086    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"]
Nov 02 20:08:35.238892 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.238865    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.244099 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.244073    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"]
Nov 02 20:08:35.276948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.276910    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.277192 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.277166    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.277312 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.277298    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.277456 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.277438    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mpzm\" (UniqueName: \"kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-kube-api-access-8mpzm\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.277664 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.277645    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.278014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.277811    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.278201 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.278181    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d522fc-4998-4ef6-a473-f6fb90335551-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.278952 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.278925    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.279041 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.278995    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.287817 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.284911    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"]
Nov 02 20:08:35.287817 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.287685    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:35.342366 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.342307    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf" event={"ID":"402ae8c8-446f-4812-9ecc-9cb29b9c35fe","Type":"ContainerStarted","Data":"fa0d25659638c0f643091b7f434e6d0d9e58219f659edff0c64e120bbd223913"}
Nov 02 20:08:35.342545 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.342390    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf" event={"ID":"402ae8c8-446f-4812-9ecc-9cb29b9c35fe","Type":"ContainerStarted","Data":"c0ace55e88124363ceb094e55819899c02af8611f07ec3e3b50c1cdf22d6c99f"}
Nov 02 20:08:35.344257 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.344227    5224 generic.go:356] "Generic (PLEG): container finished" podID="644fb6e2-8934-4031-a125-8c9fceea7397" containerID="4fb4ae8c2d23740b115eb41906a8d720c6acac8a1f2647afc391fd506e8611b7" exitCode=0
Nov 02 20:08:35.344395 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.344304    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z" event={"ID":"644fb6e2-8934-4031-a125-8c9fceea7397","Type":"ContainerDied","Data":"4fb4ae8c2d23740b115eb41906a8d720c6acac8a1f2647afc391fd506e8611b7"}
Nov 02 20:08:35.379933 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.379899    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.379933 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.379938    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.380204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.379993    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.380204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380035    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.380204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380071    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.380204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380101    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.380204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380128    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.380204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380154    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.380204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380192    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6de519-aa8e-4768-a96f-c930e7a14dad-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.380570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380240    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mpzm\" (UniqueName: \"kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-kube-api-access-8mpzm\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.380570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380280    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.380570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380313    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k44dt\" (UniqueName: \"kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-kube-api-access-k44dt\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.380570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380365    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.380570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380474    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.380570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380505    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.380570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380535    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.380570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380569    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d522fc-4998-4ef6-a473-f6fb90335551-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.381467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380605    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.381467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380635    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.381467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.380967    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.381467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.381305    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.381467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.381417    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.381940 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.381909    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d522fc-4998-4ef6-a473-f6fb90335551-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.384124 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.384079    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.385901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.385860    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.438476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.438418    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.438476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.438470    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mpzm\" (UniqueName: \"kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-kube-api-access-8mpzm\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.482218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.482181    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.482218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.482224    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.482494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.482260    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.482494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.482290    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6de519-aa8e-4768-a96f-c930e7a14dad-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.482494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.482356    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.482494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.482385    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k44dt\" (UniqueName: \"kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-kube-api-access-k44dt\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.482494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.482475    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.482756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.482523    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.482756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.482554    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.482756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.482644    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.482957 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.482910    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.483046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.483027    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.483289 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.483268    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.483588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.483562    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6de519-aa8e-4768-a96f-c930e7a14dad-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.484913 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.484894    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.484990 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.484935    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.539488 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.539457    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.540027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.540008    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k44dt\" (UniqueName: \"kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-kube-api-access-k44dt\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.546888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.546865    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:35.586663 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.586634    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:35.707938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.707870    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf" podStartSLOduration=2.707850508 podStartE2EDuration="2.707850508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:35.706190871 +0000 UTC m=+1196.703879495" watchObservedRunningTime="2026-11-02 20:08:35.707850508 +0000 UTC m=+1196.705539118"
Nov 02 20:08:35.971871 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.971140    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:35.975063 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.975020    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 20:08:35.975276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.975096    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf" podUID="402ae8c8-446f-4812-9ecc-9cb29b9c35fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 20:08:35.988751 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:08:35.988714    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d522fc_4998_4ef6_a473_f6fb90335551.slice/crio-019dd87236604c73bdf32960a4870ebfcf81b229ee2335a922340440177f96a5 WatchSource:0}: Error finding container 019dd87236604c73bdf32960a4870ebfcf81b229ee2335a922340440177f96a5: Status 404 returned error can't find the container with id 019dd87236604c73bdf32960a4870ebfcf81b229ee2335a922340440177f96a5
Nov 02 20:08:35.991392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.991353    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:08:35.991516 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.991444    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:08:35.991516 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.991498    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:08:35.992345 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:35.992320    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"]
Nov 02 20:08:36.037514 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:08:36.037478    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6de519_aa8e_4768_a96f_c930e7a14dad.slice/crio-5b554a524d189cb318db9c2da7c07092659d19510f553f9acafbe3bb3acdcc98 WatchSource:0}: Error finding container 5b554a524d189cb318db9c2da7c07092659d19510f553f9acafbe3bb3acdcc98: Status 404 returned error can't find the container with id 5b554a524d189cb318db9c2da7c07092659d19510f553f9acafbe3bb3acdcc98
Nov 02 20:08:36.040327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.040286    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:08:36.040452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.040376    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:08:36.040452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.040417    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:08:36.042090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.042031    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"]
Nov 02 20:08:36.361848 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.360375    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7" event={"ID":"b5d522fc-4998-4ef6-a473-f6fb90335551","Type":"ContainerStarted","Data":"a7b65e4d033faced083453664399e12b25a48699f3d7b5f52da5d3b527212c03"}
Nov 02 20:08:36.361848 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.360471    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7" event={"ID":"b5d522fc-4998-4ef6-a473-f6fb90335551","Type":"ContainerStarted","Data":"019dd87236604c73bdf32960a4870ebfcf81b229ee2335a922340440177f96a5"}
Nov 02 20:08:36.380883 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.379956    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67" event={"ID":"bb6de519-aa8e-4768-a96f-c930e7a14dad","Type":"ContainerStarted","Data":"42bcf710ade99afbe2cdca2241fef83d652f1fe4aba0345db71f90d27632676c"}
Nov 02 20:08:36.380883 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.380030    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67" event={"ID":"bb6de519-aa8e-4768-a96f-c930e7a14dad","Type":"ContainerStarted","Data":"5b554a524d189cb318db9c2da7c07092659d19510f553f9acafbe3bb3acdcc98"}
Nov 02 20:08:36.544985 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.544959    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"
Nov 02 20:08:36.547394 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.547380    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:36.587199 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.587155    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:36.601641 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.601584    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7" podStartSLOduration=1.60157006 podStartE2EDuration="1.60157006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:36.483208497 +0000 UTC m=+1197.480897110" watchObservedRunningTime="2026-11-02 20:08:36.60157006 +0000 UTC m=+1197.599258667"
Nov 02 20:08:36.603786 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.603728    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67" podStartSLOduration=1.603716548 podStartE2EDuration="1.603716548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:36.601635548 +0000 UTC m=+1197.599324161" watchObservedRunningTime="2026-11-02 20:08:36.603716548 +0000 UTC m=+1197.601405156"
Nov 02 20:08:36.698349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.698303    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-openshift-ca\") pod \"644fb6e2-8934-4031-a125-8c9fceea7397\" (UID: \"644fb6e2-8934-4031-a125-8c9fceea7397\") "
Nov 02 20:08:36.698714 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.698692    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644fb6e2-8934-4031-a125-8c9fceea7397-kube-api-access-465qg\" (UniqueName: \"kubernetes.io/projected/644fb6e2-8934-4031-a125-8c9fceea7397-kube-api-access-465qg\") pod \"644fb6e2-8934-4031-a125-8c9fceea7397\" (UID: \"644fb6e2-8934-4031-a125-8c9fceea7397\") "
Nov 02 20:08:36.698938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.698895    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-xks-ca\") pod \"644fb6e2-8934-4031-a125-8c9fceea7397\" (UID: \"644fb6e2-8934-4031-a125-8c9fceea7397\") "
Nov 02 20:08:36.719078 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.716932    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-openshift-ca" pod "644fb6e2-8934-4031-a125-8c9fceea7397" (UID: "644fb6e2-8934-4031-a125-8c9fceea7397"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:36.719489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.719452    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-xks-ca" pod "644fb6e2-8934-4031-a125-8c9fceea7397" (UID: "644fb6e2-8934-4031-a125-8c9fceea7397"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:08:36.724170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.723947    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644fb6e2-8934-4031-a125-8c9fceea7397-kube-api-access-465qg" pod "644fb6e2-8934-4031-a125-8c9fceea7397" (UID: "644fb6e2-8934-4031-a125-8c9fceea7397"). InnerVolumeSpecName "kube-api-access-465qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:36.801784 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.801600    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:36.801784 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.801646    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-465qg\" (UniqueName: \"kubernetes.io/projected/644fb6e2-8934-4031-a125-8c9fceea7397-kube-api-access-465qg\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:36.801784 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.801667    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644fb6e2-8934-4031-a125-8c9fceea7397-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:36.983319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.983222    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 20:08:36.983595 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:36.983565    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf" podUID="402ae8c8-446f-4812-9ecc-9cb29b9c35fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 20:08:37.419793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:37.419381    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"
Nov 02 20:08:37.424043 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:37.424008    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z" event={"ID":"644fb6e2-8934-4031-a125-8c9fceea7397","Type":"ContainerDied","Data":"cf12bb881588a3884cb4ecc6179bde5db311d4edac8d2cf74884d2e0031b2454"}
Nov 02 20:08:37.424282 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:37.424267    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf12bb881588a3884cb4ecc6179bde5db311d4edac8d2cf74884d2e0031b2454"
Nov 02 20:08:37.556960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:37.556922    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 20:08:37.557257 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:37.557226    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7" podUID="b5d522fc-4998-4ef6-a473-f6fb90335551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 20:08:37.590760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:37.590726    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 20:08:37.591020 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:37.590993    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67" podUID="bb6de519-aa8e-4768-a96f-c930e7a14dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 20:08:37.985851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:37.971168    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 20:08:37.985851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:37.971244    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf" podUID="402ae8c8-446f-4812-9ecc-9cb29b9c35fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 20:08:38.547116 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:38.547088    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 20:08:38.547564 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:38.547540    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7" podUID="b5d522fc-4998-4ef6-a473-f6fb90335551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 20:08:38.587236 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:38.587186    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 20:08:38.587593 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:38.587529    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67" podUID="bb6de519-aa8e-4768-a96f-c930e7a14dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 20:08:38.979853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:38.976272    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 20:08:38.979853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:38.976332    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf" podUID="402ae8c8-446f-4812-9ecc-9cb29b9c35fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 20:08:39.547378 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:39.547342    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 20:08:39.547881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:39.547401    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7" podUID="b5d522fc-4998-4ef6-a473-f6fb90335551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 20:08:39.598794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:39.594161    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 20:08:39.598794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:39.594225    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67" podUID="bb6de519-aa8e-4768-a96f-c930e7a14dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 20:08:39.997792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:39.997588    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 20:08:39.997792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:39.997656    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf" podUID="402ae8c8-446f-4812-9ecc-9cb29b9c35fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 20:08:40.337175 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:40.337135    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:40.547300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:40.547260    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 20:08:40.547504 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:40.547315    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7" podUID="b5d522fc-4998-4ef6-a473-f6fb90335551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 20:08:40.591380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:40.591283    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 20:08:40.591647 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:40.591613    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67" podUID="bb6de519-aa8e-4768-a96f-c930e7a14dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 20:08:40.971849 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:40.971055    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 20:08:40.971849 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:40.971152    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf" podUID="402ae8c8-446f-4812-9ecc-9cb29b9c35fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 20:08:41.547527 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:41.547482    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 20:08:41.548061 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:41.547535    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7" podUID="b5d522fc-4998-4ef6-a473-f6fb90335551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 20:08:41.594793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:41.587221    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 20:08:41.594793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:41.587282    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67" podUID="bb6de519-aa8e-4768-a96f-c930e7a14dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 20:08:42.039631 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:42.039574    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:42.214811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:42.214752    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:42.286818 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:42.286760    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:42.287902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:42.287876    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:42.288506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:42.288476    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:08:42.547757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:42.547714    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 20:08:42.548165 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:42.547813    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7" podUID="b5d522fc-4998-4ef6-a473-f6fb90335551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 20:08:42.588069 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:42.588030    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 20:08:42.588270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:42.588092    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67" podUID="bb6de519-aa8e-4768-a96f-c930e7a14dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 20:08:43.561791 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:43.561219    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:43.562267 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:43.562145    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:43.562943 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:43.562915    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:08:43.591845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:43.591812    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:43.592135 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:43.592117    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:43.592819 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:43.592796    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:08:43.713029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:43.712991    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:45.247394 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:45.247354    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:45.831785 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:45.831734    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:08:49.822150 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:49.822110    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"]
Nov 02 20:08:49.822743 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:49.822722    5224 cpu_manager.go:414] "containerMap: removing container" podUID="644fb6e2-8934-4031-a125-8c9fceea7397" containerName="revoke-keys"
Nov 02 20:08:49.822856 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:49.822745    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644fb6e2-8934-4031-a125-8c9fceea7397" containerName="revoke-keys"
Nov 02 20:08:49.822928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:49.822913    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="644fb6e2-8934-4031-a125-8c9fceea7397" containerName="revoke-keys"
Nov 02 20:08:49.827908 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:49.827888    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"
Nov 02 20:08:49.878427 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:49.878395    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"]
Nov 02 20:08:49.923699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:49.923657    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38209362-tqvfw\" (UID: \"806ccf88-3a32-4615-823a-b9db72fd51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"
Nov 02 20:08:49.924205 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:49.924143    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zct9b\" (UniqueName: \"kubernetes.io/projected/806ccf88-3a32-4615-823a-b9db72fd51f4-kube-api-access-zct9b\") pod \"maas-api-revoke-sub-38209362-tqvfw\" (UID: \"806ccf88-3a32-4615-823a-b9db72fd51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"
Nov 02 20:08:49.924491 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:49.924470    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38209362-tqvfw\" (UID: \"806ccf88-3a32-4615-823a-b9db72fd51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"
Nov 02 20:08:49.930616 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:49.930584    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:08:49.930616 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:49.930614    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:08:49.930838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:49.930629    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:08:49.932382 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:49.932364    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:08:50.026267 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:50.026234    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zct9b\" (UniqueName: \"kubernetes.io/projected/806ccf88-3a32-4615-823a-b9db72fd51f4-kube-api-access-zct9b\") pod \"maas-api-revoke-sub-38209362-tqvfw\" (UID: \"806ccf88-3a32-4615-823a-b9db72fd51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"
Nov 02 20:08:50.026477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:50.026293    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38209362-tqvfw\" (UID: \"806ccf88-3a32-4615-823a-b9db72fd51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"
Nov 02 20:08:50.026477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:50.026327    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38209362-tqvfw\" (UID: \"806ccf88-3a32-4615-823a-b9db72fd51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"
Nov 02 20:08:50.027046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:50.027018    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38209362-tqvfw\" (UID: \"806ccf88-3a32-4615-823a-b9db72fd51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"
Nov 02 20:08:50.028679 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:50.028661    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38209362-tqvfw\" (UID: \"806ccf88-3a32-4615-823a-b9db72fd51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"
Nov 02 20:08:50.084793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:50.084697    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zct9b\" (UniqueName: \"kubernetes.io/projected/806ccf88-3a32-4615-823a-b9db72fd51f4-kube-api-access-zct9b\") pod \"maas-api-revoke-sub-38209362-tqvfw\" (UID: \"806ccf88-3a32-4615-823a-b9db72fd51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"
Nov 02 20:08:50.141214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:50.141175    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"
Nov 02 20:08:50.415163 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:50.415074    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:50.509209 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:08:50.509180    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806ccf88_3a32_4615_823a_b9db72fd51f4.slice/crio-c0c513d570a9b3d6897085eb062da1d354ef22a8307027ba04f945d60e9d0159 WatchSource:0}: Error finding container c0c513d570a9b3d6897085eb062da1d354ef22a8307027ba04f945d60e9d0159: Status 404 returned error can't find the container with id c0c513d570a9b3d6897085eb062da1d354ef22a8307027ba04f945d60e9d0159
Nov 02 20:08:50.512395 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:50.512370    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"]
Nov 02 20:08:50.543345 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:50.543313    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw" event={"ID":"806ccf88-3a32-4615-823a-b9db72fd51f4","Type":"ContainerStarted","Data":"c0c513d570a9b3d6897085eb062da1d354ef22a8307027ba04f945d60e9d0159"}
Nov 02 20:08:51.420824 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:51.420786    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:51.551355 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:51.551317    5224 generic.go:356] "Generic (PLEG): container finished" podID="806ccf88-3a32-4615-823a-b9db72fd51f4" containerID="902dc2699b47cfcbc38496e7f908588be28b0e5641629a4f8d96ff19d3fbe244" exitCode=0
Nov 02 20:08:51.551549 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:51.551406    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw" event={"ID":"806ccf88-3a32-4615-823a-b9db72fd51f4","Type":"ContainerDied","Data":"902dc2699b47cfcbc38496e7f908588be28b0e5641629a4f8d96ff19d3fbe244"}
Nov 02 20:08:52.449786 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:52.449734    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:52.693476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:52.693452    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"
Nov 02 20:08:52.861244 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:52.861211    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-openshift-ca\") pod \"806ccf88-3a32-4615-823a-b9db72fd51f4\" (UID: \"806ccf88-3a32-4615-823a-b9db72fd51f4\") "
Nov 02 20:08:52.861449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:52.861266    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806ccf88-3a32-4615-823a-b9db72fd51f4-kube-api-access-zct9b\" (UniqueName: \"kubernetes.io/projected/806ccf88-3a32-4615-823a-b9db72fd51f4-kube-api-access-zct9b\") pod \"806ccf88-3a32-4615-823a-b9db72fd51f4\" (UID: \"806ccf88-3a32-4615-823a-b9db72fd51f4\") "
Nov 02 20:08:52.861449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:52.861300    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-xks-ca\") pod \"806ccf88-3a32-4615-823a-b9db72fd51f4\" (UID: \"806ccf88-3a32-4615-823a-b9db72fd51f4\") "
Nov 02 20:08:52.861690 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:52.861663    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-openshift-ca" pod "806ccf88-3a32-4615-823a-b9db72fd51f4" (UID: "806ccf88-3a32-4615-823a-b9db72fd51f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:52.861832 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:52.861816    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:52.863435 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:52.863407    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-xks-ca" pod "806ccf88-3a32-4615-823a-b9db72fd51f4" (UID: "806ccf88-3a32-4615-823a-b9db72fd51f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:08:52.863568 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:52.863513    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806ccf88-3a32-4615-823a-b9db72fd51f4-kube-api-access-zct9b" pod "806ccf88-3a32-4615-823a-b9db72fd51f4" (UID: "806ccf88-3a32-4615-823a-b9db72fd51f4"). InnerVolumeSpecName "kube-api-access-zct9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:52.963318 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:52.963278    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zct9b\" (UniqueName: \"kubernetes.io/projected/806ccf88-3a32-4615-823a-b9db72fd51f4-kube-api-access-zct9b\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:52.963318 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:52.963311    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806ccf88-3a32-4615-823a-b9db72fd51f4-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:53.564540 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:53.564503    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw" event={"ID":"806ccf88-3a32-4615-823a-b9db72fd51f4","Type":"ContainerDied","Data":"c0c513d570a9b3d6897085eb062da1d354ef22a8307027ba04f945d60e9d0159"}
Nov 02 20:08:53.564540 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:53.564522    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"
Nov 02 20:08:53.565137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:53.564553    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0c513d570a9b3d6897085eb062da1d354ef22a8307027ba04f945d60e9d0159"
Nov 02 20:08:55.276131 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:55.276092    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:56.288465 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:56.288325    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:08:57.359355 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:08:57.359320    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:08:58.496373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:08:58.496333    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:00.476692 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.476637    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:00.700387 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.700303    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"]
Nov 02 20:09:00.700860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.700841    5224 cpu_manager.go:414] "containerMap: removing container" podUID="806ccf88-3a32-4615-823a-b9db72fd51f4" containerName="revoke-keys"
Nov 02 20:09:00.700969 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.700863    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806ccf88-3a32-4615-823a-b9db72fd51f4" containerName="revoke-keys"
Nov 02 20:09:00.701033 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.701005    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="806ccf88-3a32-4615-823a-b9db72fd51f4" containerName="revoke-keys"
Nov 02 20:09:00.704441 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.704420    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"
Nov 02 20:09:00.756104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.756077    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"]
Nov 02 20:09:00.757338 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.757315    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npctr\" (UniqueName: \"kubernetes.io/projected/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-kube-api-access-npctr\") pod \"maas-api-revoke-sub-c7d53422-kldrw\" (UID: \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"
Nov 02 20:09:00.757489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.757465    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7d53422-kldrw\" (UID: \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"
Nov 02 20:09:00.757586 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.757555    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7d53422-kldrw\" (UID: \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"
Nov 02 20:09:00.804459 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.804421    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:09:00.804626 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.804484    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:09:00.804626 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.804526    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:09:00.804626 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.804538    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:09:00.858536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.858503    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7d53422-kldrw\" (UID: \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"
Nov 02 20:09:00.858536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.858539    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7d53422-kldrw\" (UID: \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"
Nov 02 20:09:00.858763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.858595    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npctr\" (UniqueName: \"kubernetes.io/projected/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-kube-api-access-npctr\") pod \"maas-api-revoke-sub-c7d53422-kldrw\" (UID: \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"
Nov 02 20:09:00.859315 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.859294    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7d53422-kldrw\" (UID: \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"
Nov 02 20:09:00.861348 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.861327    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7d53422-kldrw\" (UID: \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"
Nov 02 20:09:00.917111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:00.917086    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npctr\" (UniqueName: \"kubernetes.io/projected/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-kube-api-access-npctr\") pod \"maas-api-revoke-sub-c7d53422-kldrw\" (UID: \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"
Nov 02 20:09:01.017842 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:01.017814    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"
Nov 02 20:09:01.385051 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:09:01.385026    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6aa83bd_ffa5_4346_b9b6_2ae139b9388b.slice/crio-e273defc824baab9f095b562ca065cbd51c17a1f117bd0cd0b4f5e6e21ee2b04 WatchSource:0}: Error finding container e273defc824baab9f095b562ca065cbd51c17a1f117bd0cd0b4f5e6e21ee2b04: Status 404 returned error can't find the container with id e273defc824baab9f095b562ca065cbd51c17a1f117bd0cd0b4f5e6e21ee2b04
Nov 02 20:09:01.388310 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:01.388281    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"]
Nov 02 20:09:01.632116 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:01.632028    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw" event={"ID":"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b","Type":"ContainerStarted","Data":"e273defc824baab9f095b562ca065cbd51c17a1f117bd0cd0b4f5e6e21ee2b04"}
Nov 02 20:09:02.279323 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:09:02.279279    5224 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6aa83bd_ffa5_4346_b9b6_2ae139b9388b.slice/crio-conmon-a3a5876be327af15d0610114bd4e9cfb425d17862e06f3b1477e6b494b2fcb3d.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 20:09:02.639776 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:02.639735    5224 generic.go:356] "Generic (PLEG): container finished" podID="a6aa83bd-ffa5-4346-b9b6-2ae139b9388b" containerID="a3a5876be327af15d0610114bd4e9cfb425d17862e06f3b1477e6b494b2fcb3d" exitCode=0
Nov 02 20:09:02.640142 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:02.639813    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw" event={"ID":"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b","Type":"ContainerDied","Data":"a3a5876be327af15d0610114bd4e9cfb425d17862e06f3b1477e6b494b2fcb3d"}
Nov 02 20:09:03.789693 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:03.789664    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"
Nov 02 20:09:03.892959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:03.892925    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-kube-api-access-npctr\" (UniqueName: \"kubernetes.io/projected/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-kube-api-access-npctr\") pod \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\" (UID: \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\") "
Nov 02 20:09:03.893152 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:03.893098    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-openshift-ca\") pod \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\" (UID: \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\") "
Nov 02 20:09:03.893152 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:03.893131    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-xks-ca\") pod \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\" (UID: \"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b\") "
Nov 02 20:09:03.893512 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:03.893481    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-openshift-ca" pod "a6aa83bd-ffa5-4346-b9b6-2ae139b9388b" (UID: "a6aa83bd-ffa5-4346-b9b6-2ae139b9388b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:03.895134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:03.895107    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-xks-ca" pod "a6aa83bd-ffa5-4346-b9b6-2ae139b9388b" (UID: "a6aa83bd-ffa5-4346-b9b6-2ae139b9388b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:03.895304 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:03.895284    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-kube-api-access-npctr" pod "a6aa83bd-ffa5-4346-b9b6-2ae139b9388b" (UID: "a6aa83bd-ffa5-4346-b9b6-2ae139b9388b"). InnerVolumeSpecName "kube-api-access-npctr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:03.994357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:03.994317    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:03.994357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:03.994353    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:03.994357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:03.994366    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npctr\" (UniqueName: \"kubernetes.io/projected/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b-kube-api-access-npctr\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:04.652639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:04.652603    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw" event={"ID":"a6aa83bd-ffa5-4346-b9b6-2ae139b9388b","Type":"ContainerDied","Data":"e273defc824baab9f095b562ca065cbd51c17a1f117bd0cd0b4f5e6e21ee2b04"}
Nov 02 20:09:04.652639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:04.652631    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"
Nov 02 20:09:04.652879 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:04.652668    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e273defc824baab9f095b562ca065cbd51c17a1f117bd0cd0b4f5e6e21ee2b04"
Nov 02 20:09:05.471443 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.471404    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:05.530524 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.530490    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"]
Nov 02 20:09:05.530978 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.530964    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a6aa83bd-ffa5-4346-b9b6-2ae139b9388b" containerName="revoke-keys"
Nov 02 20:09:05.531056 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.530979    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6aa83bd-ffa5-4346-b9b6-2ae139b9388b" containerName="revoke-keys"
Nov 02 20:09:05.531098 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.531061    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6aa83bd-ffa5-4346-b9b6-2ae139b9388b" containerName="revoke-keys"
Nov 02 20:09:05.534642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.534622    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.590883 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.590845    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-14d090-openshift-default-dockercfg-5wl9l\""
Nov 02 20:09:05.611680 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.611644    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.611855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.611699    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.611855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.611802    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.611855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.611839    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.612024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.611878    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.612024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.611988    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9pvm\" (UniqueName: \"kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-kube-api-access-d9pvm\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.612148 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.612131    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.612215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.612200    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.612275 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.612256    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.644490 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.644456    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"]
Nov 02 20:09:05.713664 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.713624    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.713929 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.713671    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.713929 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.713709    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.713929 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.713741    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.713929 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.713812    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.713929 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.713871    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.713929 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.713915    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9pvm\" (UniqueName: \"kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-kube-api-access-d9pvm\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.714276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.714043    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.714276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.714096    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.714276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.714115    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.714276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.714170    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.714276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.714213    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.714511 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.714397    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.714547 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.714527    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.716258 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.716234    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.716400 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.716382    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.771792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.771728    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.771957 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.771901    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9pvm\" (UniqueName: \"kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-kube-api-access-d9pvm\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:05.848114 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:05.848081    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:06.263829 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:09:06.263802    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0279bf_c6af_4283_9db8_60d0c8dd7a44.slice/crio-97628173c2edaceb0f4749c534fcab3d6d1922831da6633ba4705a6c4ae142d6 WatchSource:0}: Error finding container 97628173c2edaceb0f4749c534fcab3d6d1922831da6633ba4705a6c4ae142d6: Status 404 returned error can't find the container with id 97628173c2edaceb0f4749c534fcab3d6d1922831da6633ba4705a6c4ae142d6
Nov 02 20:09:06.266411 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:06.266380    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:09:06.266486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:06.266446    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:09:06.266486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:06.266477    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:09:06.267969 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:06.267948    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"]
Nov 02 20:09:06.666033 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:06.665996    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" event={"ID":"6a0279bf-c6af-4283-9db8-60d0c8dd7a44","Type":"ContainerStarted","Data":"1ab9e3d0087e5d676e0a29cd44e0bebd66bd2c42f456d8818c426e3b044b8b14"}
Nov 02 20:09:06.666400 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:06.666055    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" event={"ID":"6a0279bf-c6af-4283-9db8-60d0c8dd7a44","Type":"ContainerStarted","Data":"97628173c2edaceb0f4749c534fcab3d6d1922831da6633ba4705a6c4ae142d6"}
Nov 02 20:09:06.784483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:06.784433    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" podStartSLOduration=1.784419578 podStartE2EDuration="1.784419578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:06.782966547 +0000 UTC m=+1227.780655156" watchObservedRunningTime="2026-11-02 20:09:06.784419578 +0000 UTC m=+1227.782108186"
Nov 02 20:09:06.849156 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:06.849120    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:07.850595 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:07.850552    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 20:09:07.855344 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:07.855305    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" podUID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 20:09:08.055035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:08.054993    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"]
Nov 02 20:09:08.101095 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:08.101013    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c86ff75-rcj8r"]
Nov 02 20:09:08.853791 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:08.853164    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 20:09:08.853791 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:08.853228    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" podUID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 20:09:09.180731 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:09.180647    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deceea06-2be9-48c0-b59e-68e149119bed" path="/var/lib/kubelet/pods/deceea06-2be9-48c0-b59e-68e149119bed/volumes"
Nov 02 20:09:09.848640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:09.848603    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 20:09:09.848901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:09.848668    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" podUID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 20:09:10.349595 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:10.349561    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:10.848414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:10.848382    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 20:09:10.848601 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:10.848444    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" podUID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 20:09:10.875234 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:10.875188    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:09:11.585600 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:11.585564    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:11.784863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:11.778139    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:11.848898 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:11.848757    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 20:09:11.849566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:11.849140    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" podUID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 20:09:12.851686 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:12.851197    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 20:09:12.851686 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:12.851264    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" podUID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 20:09:13.055948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:13.055893    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk" podUID="41edc495-0352-461a-a80c-c4dd7358e0c3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:13.096789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:13.092846    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:13.117141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:13.117019    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs" podUID="c50013f5-4bb1-4b40-b62f-5b736b953ac0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:13.221917 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:13.221855    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj" podUID="f4ff50fb-2482-4330-9ce4-e05d794c2a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:13.362702 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:13.362635    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6" podUID="675c9473-71e0-4bd1-8804-e684edf052fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 20:09:13.448949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:13.448852    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" podUID="87bb590b-e94c-4051-9df7-09be7673c1e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:13.859699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:13.859666    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 20:09:13.860676 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:13.860289    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" podUID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 20:09:15.110231 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:15.110198    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:15.110673 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:15.110531    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:15.111362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:15.111339    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:09:15.951094 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:15.951051    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:19.155358 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.155323    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"]
Nov 02 20:09:19.163848 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.163825    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"
Nov 02 20:09:19.211142 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.211109    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"]
Nov 02 20:09:19.258719 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.258691    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:09:19.258719 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.258719    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:09:19.258970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.258748    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:09:19.268592 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.268564    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:09:19.346438 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.346406    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmg2x\" (UniqueName: \"kubernetes.io/projected/d8d969f2-40af-4308-9062-f27f9063bc5f-kube-api-access-lmg2x\") pod \"maas-api-revoke-sub-7893d5df-52znw\" (UID: \"d8d969f2-40af-4308-9062-f27f9063bc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"
Nov 02 20:09:19.346620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.346472    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7893d5df-52znw\" (UID: \"d8d969f2-40af-4308-9062-f27f9063bc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"
Nov 02 20:09:19.346620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.346581    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7893d5df-52znw\" (UID: \"d8d969f2-40af-4308-9062-f27f9063bc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"
Nov 02 20:09:19.448089 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.447994    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmg2x\" (UniqueName: \"kubernetes.io/projected/d8d969f2-40af-4308-9062-f27f9063bc5f-kube-api-access-lmg2x\") pod \"maas-api-revoke-sub-7893d5df-52znw\" (UID: \"d8d969f2-40af-4308-9062-f27f9063bc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"
Nov 02 20:09:19.448089 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.448078    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7893d5df-52znw\" (UID: \"d8d969f2-40af-4308-9062-f27f9063bc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"
Nov 02 20:09:19.448314 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.448116    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7893d5df-52znw\" (UID: \"d8d969f2-40af-4308-9062-f27f9063bc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"
Nov 02 20:09:19.448879 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.448836    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7893d5df-52znw\" (UID: \"d8d969f2-40af-4308-9062-f27f9063bc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"
Nov 02 20:09:19.450433 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.450409    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7893d5df-52znw\" (UID: \"d8d969f2-40af-4308-9062-f27f9063bc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"
Nov 02 20:09:19.506062 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.506033    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmg2x\" (UniqueName: \"kubernetes.io/projected/d8d969f2-40af-4308-9062-f27f9063bc5f-kube-api-access-lmg2x\") pod \"maas-api-revoke-sub-7893d5df-52znw\" (UID: \"d8d969f2-40af-4308-9062-f27f9063bc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"
Nov 02 20:09:19.708899 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.708805    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:19.778670 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:19.778633    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"
Nov 02 20:09:20.153272 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:09:20.153240    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d969f2_40af_4308_9062_f27f9063bc5f.slice/crio-309a5d5ba2f2e686aadd2ead3b7e3810babd0d4b5141c40f331041ba3591795c WatchSource:0}: Error finding container 309a5d5ba2f2e686aadd2ead3b7e3810babd0d4b5141c40f331041ba3591795c: Status 404 returned error can't find the container with id 309a5d5ba2f2e686aadd2ead3b7e3810babd0d4b5141c40f331041ba3591795c
Nov 02 20:09:20.156443 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.156415    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"]
Nov 02 20:09:20.660389 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.660355    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"]
Nov 02 20:09:20.664646 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.664621    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.764828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.764794    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.764991 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.764855    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a0e46b-fc81-4841-a517-b56e0a54061d-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.764991 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.764927    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.765079 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.764987    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.765079 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.765045    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.765165 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.765109    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.765211 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.765189    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.765252 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.765228    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.765306 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.765257    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drbhb\" (UniqueName: \"kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-kube-api-access-drbhb\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.775276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.775248    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"]
Nov 02 20:09:20.831994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.831960    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw" event={"ID":"d8d969f2-40af-4308-9062-f27f9063bc5f","Type":"ContainerStarted","Data":"4d9956fd6d2844da771881c98a618c9571ef3160fb163deb20d05b221150ec6a"}
Nov 02 20:09:20.832141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.832012    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw" event={"ID":"d8d969f2-40af-4308-9062-f27f9063bc5f","Type":"ContainerStarted","Data":"309a5d5ba2f2e686aadd2ead3b7e3810babd0d4b5141c40f331041ba3591795c"}
Nov 02 20:09:20.866473 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.866441    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.866473 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.866472    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.866762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.866504    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.866762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.866524    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.866762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.866545    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.866762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.866576    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbhb\" (UniqueName: \"kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-kube-api-access-drbhb\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.866762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.866635    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.866762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.866683    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a0e46b-fc81-4841-a517-b56e0a54061d-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.866762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.866762    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.867126 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.866853    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.867126 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.866934    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.867126 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.866978    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.867126 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.867115    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.867481 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.867464    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a0e46b-fc81-4841-a517-b56e0a54061d-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.868894 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.868872    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.869085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.869069    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.924557 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.924528    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.924835 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.924812    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbhb\" (UniqueName: \"kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-kube-api-access-drbhb\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:20.947905 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.947857    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw" podStartSLOduration=1.9478447509999999 podStartE2EDuration="1.947844751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:20.946806947 +0000 UTC m=+1241.944495553" watchObservedRunningTime="2026-11-02 20:09:20.947844751 +0000 UTC m=+1241.945533360"
Nov 02 20:09:20.977902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:20.977872    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:21.059293 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:21.059248    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:21.394898 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:09:21.394867    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a0e46b_fc81_4841_a517_b56e0a54061d.slice/crio-2e74e799878a90cb1b1878cd5f72a72a873a969b91d5d222b4ef5ec6f3389872 WatchSource:0}: Error finding container 2e74e799878a90cb1b1878cd5f72a72a873a969b91d5d222b4ef5ec6f3389872: Status 404 returned error can't find the container with id 2e74e799878a90cb1b1878cd5f72a72a873a969b91d5d222b4ef5ec6f3389872
Nov 02 20:09:21.397531 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:21.397496    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:09:21.397625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:21.397607    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:09:21.399982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:21.398119    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:09:21.402165 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:21.400762    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"]
Nov 02 20:09:21.839848 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:21.839811    5224 generic.go:356] "Generic (PLEG): container finished" podID="d8d969f2-40af-4308-9062-f27f9063bc5f" containerID="4d9956fd6d2844da771881c98a618c9571ef3160fb163deb20d05b221150ec6a" exitCode=0
Nov 02 20:09:21.840051 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:21.839904    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw" event={"ID":"d8d969f2-40af-4308-9062-f27f9063bc5f","Type":"ContainerDied","Data":"4d9956fd6d2844da771881c98a618c9571ef3160fb163deb20d05b221150ec6a"}
Nov 02 20:09:21.841665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:21.841639    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd" event={"ID":"98a0e46b-fc81-4841-a517-b56e0a54061d","Type":"ContainerStarted","Data":"9496cc314281076e4994364dd1e88a90e3e64b73ec6782c7782e5b88453ac809"}
Nov 02 20:09:21.841795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:21.841691    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd" event={"ID":"98a0e46b-fc81-4841-a517-b56e0a54061d","Type":"ContainerStarted","Data":"2e74e799878a90cb1b1878cd5f72a72a873a969b91d5d222b4ef5ec6f3389872"}
Nov 02 20:09:21.978006 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:21.977963    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:22.090246 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:22.090124    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd" podStartSLOduration=2.090104746 podStartE2EDuration="2.090104746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:22.088028847 +0000 UTC m=+1243.085717450" watchObservedRunningTime="2026-11-02 20:09:22.090104746 +0000 UTC m=+1243.087793356"
Nov 02 20:09:22.984444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:22.982941    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 20:09:22.984444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:22.982995    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd" podUID="98a0e46b-fc81-4841-a517-b56e0a54061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 20:09:23.027431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.027405    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"
Nov 02 20:09:23.094337 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.094300    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-openshift-ca\") pod \"d8d969f2-40af-4308-9062-f27f9063bc5f\" (UID: \"d8d969f2-40af-4308-9062-f27f9063bc5f\") "
Nov 02 20:09:23.094507 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.094349    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-xks-ca\") pod \"d8d969f2-40af-4308-9062-f27f9063bc5f\" (UID: \"d8d969f2-40af-4308-9062-f27f9063bc5f\") "
Nov 02 20:09:23.094587 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.094559    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d969f2-40af-4308-9062-f27f9063bc5f-kube-api-access-lmg2x\" (UniqueName: \"kubernetes.io/projected/d8d969f2-40af-4308-9062-f27f9063bc5f-kube-api-access-lmg2x\") pod \"d8d969f2-40af-4308-9062-f27f9063bc5f\" (UID: \"d8d969f2-40af-4308-9062-f27f9063bc5f\") "
Nov 02 20:09:23.094695 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.094651    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-openshift-ca" pod "d8d969f2-40af-4308-9062-f27f9063bc5f" (UID: "d8d969f2-40af-4308-9062-f27f9063bc5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:23.094939 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.094923    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:23.096483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.096460    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-xks-ca" pod "d8d969f2-40af-4308-9062-f27f9063bc5f" (UID: "d8d969f2-40af-4308-9062-f27f9063bc5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:23.096554 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.096509    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d969f2-40af-4308-9062-f27f9063bc5f-kube-api-access-lmg2x" pod "d8d969f2-40af-4308-9062-f27f9063bc5f" (UID: "d8d969f2-40af-4308-9062-f27f9063bc5f"). InnerVolumeSpecName "kube-api-access-lmg2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:23.196104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.196065    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d969f2-40af-4308-9062-f27f9063bc5f-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:23.196104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.196105    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmg2x\" (UniqueName: \"kubernetes.io/projected/d8d969f2-40af-4308-9062-f27f9063bc5f-kube-api-access-lmg2x\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:23.875482 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.875448    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"
Nov 02 20:09:23.875482 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.875474    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw" event={"ID":"d8d969f2-40af-4308-9062-f27f9063bc5f","Type":"ContainerDied","Data":"309a5d5ba2f2e686aadd2ead3b7e3810babd0d4b5141c40f331041ba3591795c"}
Nov 02 20:09:23.875758 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.875520    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="309a5d5ba2f2e686aadd2ead3b7e3810babd0d4b5141c40f331041ba3591795c"
Nov 02 20:09:23.978510 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.978474    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 20:09:23.978691 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:23.978530    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd" podUID="98a0e46b-fc81-4841-a517-b56e0a54061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 20:09:24.978214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:24.978172    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 20:09:24.978614 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:24.978238    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd" podUID="98a0e46b-fc81-4841-a517-b56e0a54061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 20:09:25.978370 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:25.978332    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 20:09:25.978797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:25.978394    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd" podUID="98a0e46b-fc81-4841-a517-b56e0a54061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 20:09:25.990201 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:25.990170    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:26.978363 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:26.978320    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 20:09:26.978532 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:26.978378    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd" podUID="98a0e46b-fc81-4841-a517-b56e0a54061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 20:09:27.377814 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:09:27.377743    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:09:27.978962 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:27.978919    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 20:09:27.979454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:27.978970    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd" podUID="98a0e46b-fc81-4841-a517-b56e0a54061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 20:09:28.982983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:28.982950    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:28.983464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:28.983329    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:28.983980 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:28.983961    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:09:30.851804 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:30.851743    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:30.913673 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:30.913633    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"]
Nov 02 20:09:30.914224 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:30.914205    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d8d969f2-40af-4308-9062-f27f9063bc5f" containerName="revoke-keys"
Nov 02 20:09:30.914323 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:30.914226    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8d969f2-40af-4308-9062-f27f9063bc5f" containerName="revoke-keys"
Nov 02 20:09:30.914378 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:30.914335    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8d969f2-40af-4308-9062-f27f9063bc5f" containerName="revoke-keys"
Nov 02 20:09:30.918136 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:30.918110    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"
Nov 02 20:09:30.970807 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:30.970742    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"]
Nov 02 20:09:30.982247 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:30.982211    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0739db3a-t7z9m\" (UID: \"349a7a77-2ded-45b4-932f-493758b431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"
Nov 02 20:09:30.982441 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:30.982266    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0739db3a-t7z9m\" (UID: \"349a7a77-2ded-45b4-932f-493758b431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"
Nov 02 20:09:30.982514 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:30.982429    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgxzp\" (UniqueName: \"kubernetes.io/projected/349a7a77-2ded-45b4-932f-493758b431a5-kube-api-access-dgxzp\") pod \"maas-api-revoke-sub-0739db3a-t7z9m\" (UID: \"349a7a77-2ded-45b4-932f-493758b431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"
Nov 02 20:09:31.018558 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.018523    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:09:31.018558 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.018533    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:09:31.018558 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.018559    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:09:31.022621 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.022592    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:09:31.083565 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.083521    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0739db3a-t7z9m\" (UID: \"349a7a77-2ded-45b4-932f-493758b431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"
Nov 02 20:09:31.083565 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.083570    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0739db3a-t7z9m\" (UID: \"349a7a77-2ded-45b4-932f-493758b431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"
Nov 02 20:09:31.083869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.083626    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgxzp\" (UniqueName: \"kubernetes.io/projected/349a7a77-2ded-45b4-932f-493758b431a5-kube-api-access-dgxzp\") pod \"maas-api-revoke-sub-0739db3a-t7z9m\" (UID: \"349a7a77-2ded-45b4-932f-493758b431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"
Nov 02 20:09:31.084222 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.084200    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0739db3a-t7z9m\" (UID: \"349a7a77-2ded-45b4-932f-493758b431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"
Nov 02 20:09:31.086018 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.085995    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0739db3a-t7z9m\" (UID: \"349a7a77-2ded-45b4-932f-493758b431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"
Nov 02 20:09:31.141579 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.141501    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgxzp\" (UniqueName: \"kubernetes.io/projected/349a7a77-2ded-45b4-932f-493758b431a5-kube-api-access-dgxzp\") pod \"maas-api-revoke-sub-0739db3a-t7z9m\" (UID: \"349a7a77-2ded-45b4-932f-493758b431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"
Nov 02 20:09:31.237439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.237409    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"
Nov 02 20:09:31.603836 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:09:31.603803    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349a7a77_2ded_45b4_932f_493758b431a5.slice/crio-fe7d37cc459b9c0be41aac9cbb342dcc6fe4ed2ba03aa23fc5e80917fb91ca70 WatchSource:0}: Error finding container fe7d37cc459b9c0be41aac9cbb342dcc6fe4ed2ba03aa23fc5e80917fb91ca70: Status 404 returned error can't find the container with id fe7d37cc459b9c0be41aac9cbb342dcc6fe4ed2ba03aa23fc5e80917fb91ca70
Nov 02 20:09:31.607373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.607349    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"]
Nov 02 20:09:31.932920 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.932885    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m" event={"ID":"349a7a77-2ded-45b4-932f-493758b431a5","Type":"ContainerStarted","Data":"f92f87eb3d49704cd22f42d4ae7f3d2f1a4f1320e14a05462c989369eef612d7"}
Nov 02 20:09:31.933363 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:31.932951    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m" event={"ID":"349a7a77-2ded-45b4-932f-493758b431a5","Type":"ContainerStarted","Data":"fe7d37cc459b9c0be41aac9cbb342dcc6fe4ed2ba03aa23fc5e80917fb91ca70"}
Nov 02 20:09:32.050034 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:32.049979    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m" podStartSLOduration=2.04996629 podStartE2EDuration="2.04996629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:32.047760748 +0000 UTC m=+1253.045449357" watchObservedRunningTime="2026-11-02 20:09:32.04996629 +0000 UTC m=+1253.047654898"
Nov 02 20:09:32.942007 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:32.941972    5224 generic.go:356] "Generic (PLEG): container finished" podID="349a7a77-2ded-45b4-932f-493758b431a5" containerID="f92f87eb3d49704cd22f42d4ae7f3d2f1a4f1320e14a05462c989369eef612d7" exitCode=0
Nov 02 20:09:32.942432 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:32.942063    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m" event={"ID":"349a7a77-2ded-45b4-932f-493758b431a5","Type":"ContainerDied","Data":"f92f87eb3d49704cd22f42d4ae7f3d2f1a4f1320e14a05462c989369eef612d7"}
Nov 02 20:09:34.090620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.090590    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"]
Nov 02 20:09:34.094847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.094824    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.095577 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.095558    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"
Nov 02 20:09:34.217642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.217551    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/349a7a77-2ded-45b4-932f-493758b431a5-kube-api-access-dgxzp\" (UniqueName: \"kubernetes.io/projected/349a7a77-2ded-45b4-932f-493758b431a5-kube-api-access-dgxzp\") pod \"349a7a77-2ded-45b4-932f-493758b431a5\" (UID: \"349a7a77-2ded-45b4-932f-493758b431a5\") "
Nov 02 20:09:34.218118 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.218093    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-xks-ca\") pod \"349a7a77-2ded-45b4-932f-493758b431a5\" (UID: \"349a7a77-2ded-45b4-932f-493758b431a5\") "
Nov 02 20:09:34.218544 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.218305    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-openshift-ca\") pod \"349a7a77-2ded-45b4-932f-493758b431a5\" (UID: \"349a7a77-2ded-45b4-932f-493758b431a5\") "
Nov 02 20:09:34.218544 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.218478    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.218716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.218614    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.218716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.218660    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.218716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.218692    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.218924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.218796    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.218924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.218845    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.218924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.218886    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.218924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.218912    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7br5\" (UniqueName: \"kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-kube-api-access-x7br5\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.219568 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.219447    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.219851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.219823    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-openshift-ca" pod "349a7a77-2ded-45b4-932f-493758b431a5" (UID: "349a7a77-2ded-45b4-932f-493758b431a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:34.221218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.221196    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349a7a77-2ded-45b4-932f-493758b431a5-kube-api-access-dgxzp" pod "349a7a77-2ded-45b4-932f-493758b431a5" (UID: "349a7a77-2ded-45b4-932f-493758b431a5"). InnerVolumeSpecName "kube-api-access-dgxzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:34.221430 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.221415    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-xks-ca" pod "349a7a77-2ded-45b4-932f-493758b431a5" (UID: "349a7a77-2ded-45b4-932f-493758b431a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:34.297637 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.297602    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"]
Nov 02 20:09:34.321078 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321029    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.321279 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321099    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.321279 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321144    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.321279 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321170    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7br5\" (UniqueName: \"kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-kube-api-access-x7br5\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.321458 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321298    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.321548 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321509    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.321548 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321534    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.321660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321543    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.321660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321610    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.321660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321651    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.321851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321722    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.321851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321815    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.321959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321860    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgxzp\" (UniqueName: \"kubernetes.io/projected/349a7a77-2ded-45b4-932f-493758b431a5-kube-api-access-dgxzp\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:34.321959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321881    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:34.321959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321900    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/349a7a77-2ded-45b4-932f-493758b431a5-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:34.321959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.321904    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.322690 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.322665    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.324479 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.324435    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.325542 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.325497    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.380177 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.380143    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.380323 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.380210    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7br5\" (UniqueName: \"kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-kube-api-access-x7br5\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.408481 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.408442    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:34.826985 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:09:34.826942    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f0d2b1_4f9d_41f1_94a4_c7903a2d6ece.slice/crio-2842d742af37b5f957a63691cfb8efbde4bed4e4df39aba7107ec661c0d1cfb4 WatchSource:0}: Error finding container 2842d742af37b5f957a63691cfb8efbde4bed4e4df39aba7107ec661c0d1cfb4: Status 404 returned error can't find the container with id 2842d742af37b5f957a63691cfb8efbde4bed4e4df39aba7107ec661c0d1cfb4
Nov 02 20:09:34.829479 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.829436    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:09:34.829569 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.829524    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:09:34.829606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.829589    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:09:34.830890 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.830869    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"]
Nov 02 20:09:34.955926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.955897    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"
Nov 02 20:09:34.955926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.955920    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m" event={"ID":"349a7a77-2ded-45b4-932f-493758b431a5","Type":"ContainerDied","Data":"fe7d37cc459b9c0be41aac9cbb342dcc6fe4ed2ba03aa23fc5e80917fb91ca70"}
Nov 02 20:09:34.956191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.955972    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe7d37cc459b9c0be41aac9cbb342dcc6fe4ed2ba03aa23fc5e80917fb91ca70"
Nov 02 20:09:34.957875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.957839    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" event={"ID":"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece","Type":"ContainerStarted","Data":"9ba7033c857e6aea53a525b1c07f53d800e7edda3fe629fb0dea78adecb615f1"}
Nov 02 20:09:34.958022 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:34.957891    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" event={"ID":"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece","Type":"ContainerStarted","Data":"2842d742af37b5f957a63691cfb8efbde4bed4e4df39aba7107ec661c0d1cfb4"}
Nov 02 20:09:35.078000 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:35.077894    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" podStartSLOduration=1.077875301 podStartE2EDuration="1.077875301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:35.074835678 +0000 UTC m=+1256.072524289" watchObservedRunningTime="2026-11-02 20:09:35.077875301 +0000 UTC m=+1256.075563910"
Nov 02 20:09:35.409014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:35.408927    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:35.410640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:35.410611    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 20:09:35.410784 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:35.410660    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 20:09:35.976193 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:35.975673    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:36.409325 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:36.409289    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 20:09:36.409724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:36.409347    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 20:09:37.411262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:37.411220    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 20:09:37.411981 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:37.411931    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 20:09:38.414793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:38.414182    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 20:09:38.414793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:38.414266    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 20:09:38.927928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:38.927871    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6" podUID="f67432d1-22af-458b-bd57-002a02c62a5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:38.936921 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:38.936869    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh" podUID="3359dc52-961f-43fd-b567-3bcb22271302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:39.123846 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:39.122864    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:39.128166 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:39.126561    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs" podUID="c50013f5-4bb1-4b40-b62f-5b736b953ac0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 20:09:39.146163 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:39.146118    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk" podUID="41edc495-0352-461a-a80c-c4dd7358e0c3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:39.220455 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:39.220347    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj" podUID="f4ff50fb-2482-4330-9ce4-e05d794c2a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:39.387061 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:39.385939    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6" podUID="675c9473-71e0-4bd1-8804-e684edf052fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:39.411466 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:39.411432    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 20:09:39.411877 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:39.411847    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 20:09:39.990794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:39.928542    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6" podUID="f67432d1-22af-458b-bd57-002a02c62a5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:39.990794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:39.929082    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:09:39.990794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:39.978978    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh" podUID="3359dc52-961f-43fd-b567-3bcb22271302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:39.990794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:39.979423    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:09:40.042700 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.042655    5224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:40.160794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.123952    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:40.160794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.124727    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:09:40.160794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.133363    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs" podUID="c50013f5-4bb1-4b40-b62f-5b736b953ac0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:40.160794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.133757    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:09:40.259851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.252228    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj" podUID="f4ff50fb-2482-4330-9ce4-e05d794c2a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 20:09:40.259851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.252626    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:09:40.406794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.403439    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6" podUID="675c9473-71e0-4bd1-8804-e684edf052fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:40.430725 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.415796    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:09:40.430725 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.417069    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 20:09:40.430725 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.417109    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 20:09:40.451803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.450935    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" podUID="87bb590b-e94c-4051-9df7-09be7673c1e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:40.471628 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.470915    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:09:40.471628 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.470980    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" podUID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:40.576792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.573559    5224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6" podUID="675c9473-71e0-4bd1-8804-e684edf052fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:40.641048 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.602062    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:09:40.820486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.818303    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:09:41.000353 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:40.999233    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh" podUID="3359dc52-961f-43fd-b567-3bcb22271302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:41.035141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:41.035107    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 20:09:41.082803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:41.081995    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:09:41.141797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:41.138362    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:09:41.220563 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:41.219637    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:09:41.408864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:41.408827    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 20:09:41.409070 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:41.408892    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 20:09:42.377327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:42.377290    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:42.409529 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:42.409496    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 20:09:42.409700 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:42.409560    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 20:09:43.409850 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:43.409809    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 20:09:43.410270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:43.409871    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 20:09:43.660460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:43.660357    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:44.264626 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:44.264580    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:44.659564 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:44.659463    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:44.844107 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:44.844068    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:45.448957 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.448918    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:09:45.449224 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.449198    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:09:45.554318 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.554277    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:45.554732 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.554712    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:45.555491 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.555467    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:09:45.764969 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.764939    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"]
Nov 02 20:09:45.765421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.765406    5224 cpu_manager.go:414] "containerMap: removing container" podUID="349a7a77-2ded-45b4-932f-493758b431a5" containerName="revoke-keys"
Nov 02 20:09:45.765462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.765422    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="349a7a77-2ded-45b4-932f-493758b431a5" containerName="revoke-keys"
Nov 02 20:09:45.765536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.765527    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="349a7a77-2ded-45b4-932f-493758b431a5" containerName="revoke-keys"
Nov 02 20:09:45.770276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.770255    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"
Nov 02 20:09:45.826075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.826034    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:09:45.826263 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.826153    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:09:45.828724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.828690    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"]
Nov 02 20:09:45.837096 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.837061    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:09:45.837255 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.837148    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:09:45.840779 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.840729    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld429\" (UniqueName: \"kubernetes.io/projected/9ef576c8-8431-42a0-831b-1d87de8d8ed6-kube-api-access-ld429\") pod \"maas-api-revoke-sub-f7f58366-m9wn5\" (UID: \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"
Nov 02 20:09:45.840930 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.840800    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7f58366-m9wn5\" (UID: \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"
Nov 02 20:09:45.841014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.840990    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7f58366-m9wn5\" (UID: \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"
Nov 02 20:09:45.942563 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.942529    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7f58366-m9wn5\" (UID: \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"
Nov 02 20:09:45.942747 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.942646    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld429\" (UniqueName: \"kubernetes.io/projected/9ef576c8-8431-42a0-831b-1d87de8d8ed6-kube-api-access-ld429\") pod \"maas-api-revoke-sub-f7f58366-m9wn5\" (UID: \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"
Nov 02 20:09:45.942747 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.942679    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7f58366-m9wn5\" (UID: \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"
Nov 02 20:09:45.943306 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.943276    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7f58366-m9wn5\" (UID: \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"
Nov 02 20:09:45.945160 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:45.945136    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7f58366-m9wn5\" (UID: \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"
Nov 02 20:09:46.001403 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:46.001374    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld429\" (UniqueName: \"kubernetes.io/projected/9ef576c8-8431-42a0-831b-1d87de8d8ed6-kube-api-access-ld429\") pod \"maas-api-revoke-sub-f7f58366-m9wn5\" (UID: \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"
Nov 02 20:09:46.084571 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:46.084474    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"
Nov 02 20:09:46.453230 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:09:46.453197    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef576c8_8431_42a0_831b_1d87de8d8ed6.slice/crio-9fb0cb007a9ed8a07055ce174175af46c645961030e638e17f2c622589423f50 WatchSource:0}: Error finding container 9fb0cb007a9ed8a07055ce174175af46c645961030e638e17f2c622589423f50: Status 404 returned error can't find the container with id 9fb0cb007a9ed8a07055ce174175af46c645961030e638e17f2c622589423f50
Nov 02 20:09:46.457369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:46.457345    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"]
Nov 02 20:09:47.455550 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:47.455517    5224 generic.go:356] "Generic (PLEG): container finished" podID="9ef576c8-8431-42a0-831b-1d87de8d8ed6" containerID="61a8ff520772e0a4e482d5cbbafa09f3009b2f3cbf4750b8b752f59b4e7c9f9e" exitCode=0
Nov 02 20:09:47.456073 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:47.455587    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5" event={"ID":"9ef576c8-8431-42a0-831b-1d87de8d8ed6","Type":"ContainerDied","Data":"61a8ff520772e0a4e482d5cbbafa09f3009b2f3cbf4750b8b752f59b4e7c9f9e"}
Nov 02 20:09:47.456073 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:47.455649    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5" event={"ID":"9ef576c8-8431-42a0-831b-1d87de8d8ed6","Type":"ContainerStarted","Data":"9fb0cb007a9ed8a07055ce174175af46c645961030e638e17f2c622589423f50"}
Nov 02 20:09:47.486474 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:47.486440    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:48.615567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:48.615536    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"
Nov 02 20:09:48.780006 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:48.779972    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-openshift-ca\") pod \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\" (UID: \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\") "
Nov 02 20:09:48.780208 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:48.780063    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef576c8-8431-42a0-831b-1d87de8d8ed6-kube-api-access-ld429\" (UniqueName: \"kubernetes.io/projected/9ef576c8-8431-42a0-831b-1d87de8d8ed6-kube-api-access-ld429\") pod \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\" (UID: \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\") "
Nov 02 20:09:48.780208 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:48.780186    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-xks-ca\") pod \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\" (UID: \"9ef576c8-8431-42a0-831b-1d87de8d8ed6\") "
Nov 02 20:09:48.780414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:48.780387    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-openshift-ca" pod "9ef576c8-8431-42a0-831b-1d87de8d8ed6" (UID: "9ef576c8-8431-42a0-831b-1d87de8d8ed6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:48.780637 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:48.780620    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:48.782142 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:48.782115    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef576c8-8431-42a0-831b-1d87de8d8ed6-kube-api-access-ld429" pod "9ef576c8-8431-42a0-831b-1d87de8d8ed6" (UID: "9ef576c8-8431-42a0-831b-1d87de8d8ed6"). InnerVolumeSpecName "kube-api-access-ld429". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:48.782241 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:48.782204    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-xks-ca" pod "9ef576c8-8431-42a0-831b-1d87de8d8ed6" (UID: "9ef576c8-8431-42a0-831b-1d87de8d8ed6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:48.807320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:48.807283    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:48.883310 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:48.883273    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef576c8-8431-42a0-831b-1d87de8d8ed6-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:48.883310 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:48.883306    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld429\" (UniqueName: \"kubernetes.io/projected/9ef576c8-8431-42a0-831b-1d87de8d8ed6-kube-api-access-ld429\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:49.044245 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:49.044158    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:49.469176 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:49.469093    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"
Nov 02 20:09:49.469176 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:49.469134    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5" event={"ID":"9ef576c8-8431-42a0-831b-1d87de8d8ed6","Type":"ContainerDied","Data":"9fb0cb007a9ed8a07055ce174175af46c645961030e638e17f2c622589423f50"}
Nov 02 20:09:49.469371 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:49.469213    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fb0cb007a9ed8a07055ce174175af46c645961030e638e17f2c622589423f50"
Nov 02 20:09:51.751346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:51.751307    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:51.970902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:51.970845    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:52.149233 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.149181    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:52.498210 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.498129    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"]
Nov 02 20:09:52.498651 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.498635    5224 cpu_manager.go:414] "containerMap: removing container" podUID="9ef576c8-8431-42a0-831b-1d87de8d8ed6" containerName="revoke-keys"
Nov 02 20:09:52.498707 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.498652    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef576c8-8431-42a0-831b-1d87de8d8ed6" containerName="revoke-keys"
Nov 02 20:09:52.498758 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.498749    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef576c8-8431-42a0-831b-1d87de8d8ed6" containerName="revoke-keys"
Nov 02 20:09:52.502541 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.502515    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"
Nov 02 20:09:52.536961 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.536920    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:52.557057 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.557023    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"]
Nov 02 20:09:52.605300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.605264    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5e04421b\""
Nov 02 20:09:52.605463 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.605396    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5e04421b-dockercfg-qsbhl\""
Nov 02 20:09:52.627958 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.627917    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psn49\" (UniqueName: \"kubernetes.io/projected/20d012b3-b7bc-45da-99c6-741d3d2c01cd-kube-api-access-psn49\") pod \"payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb\" (UID: \"20d012b3-b7bc-45da-99c6-741d3d2c01cd\") " pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"
Nov 02 20:09:52.628134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.627965    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d012b3-b7bc-45da-99c6-741d3d2c01cd-plugins-config-volume\") pod \"payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb\" (UID: \"20d012b3-b7bc-45da-99c6-741d3d2c01cd\") " pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"
Nov 02 20:09:52.728989 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.728943    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psn49\" (UniqueName: \"kubernetes.io/projected/20d012b3-b7bc-45da-99c6-741d3d2c01cd-kube-api-access-psn49\") pod \"payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb\" (UID: \"20d012b3-b7bc-45da-99c6-741d3d2c01cd\") " pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"
Nov 02 20:09:52.729198 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.728998    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d012b3-b7bc-45da-99c6-741d3d2c01cd-plugins-config-volume\") pod \"payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb\" (UID: \"20d012b3-b7bc-45da-99c6-741d3d2c01cd\") " pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"
Nov 02 20:09:52.729756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.729735    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d012b3-b7bc-45da-99c6-741d3d2c01cd-plugins-config-volume\") pod \"payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb\" (UID: \"20d012b3-b7bc-45da-99c6-741d3d2c01cd\") " pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"
Nov 02 20:09:52.787234 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.787198    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psn49\" (UniqueName: \"kubernetes.io/projected/20d012b3-b7bc-45da-99c6-741d3d2c01cd-kube-api-access-psn49\") pod \"payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb\" (UID: \"20d012b3-b7bc-45da-99c6-741d3d2c01cd\") " pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"
Nov 02 20:09:52.832606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.832567    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"
Nov 02 20:09:52.983069 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:52.983033    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:53.202875 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:09:53.202830    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d012b3_b7bc_45da_99c6_741d3d2c01cd.slice/crio-2b06d021c0590b481c898d3f58606a0ecdd3701951fcbf1b42355632544f02fe WatchSource:0}: Error finding container 2b06d021c0590b481c898d3f58606a0ecdd3701951fcbf1b42355632544f02fe: Status 404 returned error can't find the container with id 2b06d021c0590b481c898d3f58606a0ecdd3701951fcbf1b42355632544f02fe
Nov 02 20:09:53.206222 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:53.206199    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"]
Nov 02 20:09:53.501346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:53.501244    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb" event={"ID":"20d012b3-b7bc-45da-99c6-741d3d2c01cd","Type":"ContainerStarted","Data":"552d93288cc854de6ead763a01924f1b6331865f74910f7f1205f25f54e60f54"}
Nov 02 20:09:53.501346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:53.501300    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb" event={"ID":"20d012b3-b7bc-45da-99c6-741d3d2c01cd","Type":"ContainerStarted","Data":"2b06d021c0590b481c898d3f58606a0ecdd3701951fcbf1b42355632544f02fe"}
Nov 02 20:09:53.501574 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:53.501352    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"
Nov 02 20:09:53.617784 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:53.617717    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb" podStartSLOduration=1.6177020500000001 podStartE2EDuration="1.61770205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:53.615734618 +0000 UTC m=+1274.613423227" watchObservedRunningTime="2026-11-02 20:09:53.61770205 +0000 UTC m=+1274.615390684"
Nov 02 20:09:55.480788 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:55.480732    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:55.659547 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:55.659515    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:55.924984 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:55.924954    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"]
Nov 02 20:09:55.928992 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:55.928966    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:55.945298 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:55.945261    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:56.038203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.038165    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"]
Nov 02 20:09:56.069821 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.069788    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21980784-069f-41e3-ae48-40cd1c32186c-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.070026 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.069857    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.070026 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.069944    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.070026 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.070016    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21980784-069f-41e3-ae48-40cd1c32186c-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.070174 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.070113    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.070174 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.070167    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.070286 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.070217    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rxgs\" (UniqueName: \"kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-kube-api-access-4rxgs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.070336 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.070296    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.070388 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.070368    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.171527 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.171487    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.171694 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.171576    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21980784-069f-41e3-ae48-40cd1c32186c-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.171694 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.171644    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.171694 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.171668    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.171882 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.171723    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21980784-069f-41e3-ae48-40cd1c32186c-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.171882 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.171788    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.171882 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.171831    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.172043 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.171879    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxgs\" (UniqueName: \"kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-kube-api-access-4rxgs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.172043 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.171907    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.172506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.172448    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.173128 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.173099    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21980784-069f-41e3-ae48-40cd1c32186c-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.173233 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.173122    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.173233 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.173104    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.173327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.173303    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.174725 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.174698    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.182823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.182711    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21980784-069f-41e3-ae48-40cd1c32186c-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.229937 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.229911    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxgs\" (UniqueName: \"kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-kube-api-access-4rxgs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.230027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.229954    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.247301 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.247276    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:56.666026 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:09:56.665986    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21980784_069f_41e3_ae48_40cd1c32186c.slice/crio-12728d4081c6cc6f5ed28ef724792b010ae73e735de0df7924549fac6030821d WatchSource:0}: Error finding container 12728d4081c6cc6f5ed28ef724792b010ae73e735de0df7924549fac6030821d: Status 404 returned error can't find the container with id 12728d4081c6cc6f5ed28ef724792b010ae73e735de0df7924549fac6030821d
Nov 02 20:09:56.669113 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.669072    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:09:56.669202 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.669157    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:09:56.669237 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.669199    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:09:56.670112 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:56.670090    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"]
Nov 02 20:09:57.384346 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:09:57.384300    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:09:57.528412 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:57.528371    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" event={"ID":"21980784-069f-41e3-ae48-40cd1c32186c","Type":"ContainerStarted","Data":"dec755234b67b5dac7818696adc61641efc8bf76a54ec3f03a54fb6ba82a0305"}
Nov 02 20:09:57.528569 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:57.528441    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" event={"ID":"21980784-069f-41e3-ae48-40cd1c32186c","Type":"ContainerStarted","Data":"12728d4081c6cc6f5ed28ef724792b010ae73e735de0df7924549fac6030821d"}
Nov 02 20:09:57.698178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:57.698082    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" podStartSLOduration=2.698068034 podStartE2EDuration="2.698068034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:57.697514521 +0000 UTC m=+1278.695203144" watchObservedRunningTime="2026-11-02 20:09:57.698068034 +0000 UTC m=+1278.695756644"
Nov 02 20:09:58.116149 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:58.116113    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:09:58.247471 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:58.247440    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:09:58.248845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:58.248818    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 20:09:58.249061 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:58.249033    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" podUID="21980784-069f-41e3-ae48-40cd1c32186c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 20:09:59.247839 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:59.247808    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 20:09:59.248216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:09:59.247869    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" podUID="21980784-069f-41e3-ae48-40cd1c32186c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 20:10:00.248562 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:00.248521    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 20:10:00.249134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:00.248588    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" podUID="21980784-069f-41e3-ae48-40cd1c32186c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 20:10:01.248309 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:01.248271    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 20:10:01.248475 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:01.248325    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" podUID="21980784-069f-41e3-ae48-40cd1c32186c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 20:10:01.575009 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:01.574970    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:10:02.249303 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:02.249271    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 20:10:02.249733 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:02.249696    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" podUID="21980784-069f-41e3-ae48-40cd1c32186c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 20:10:03.248104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:03.247653    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 20:10:03.248104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:03.247708    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" podUID="21980784-069f-41e3-ae48-40cd1c32186c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 20:10:04.002472 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:04.002432    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:04.060803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:04.059922    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk" podUID="41edc495-0352-461a-a80c-c4dd7358e0c3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:04.100794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:04.098920    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs" podUID="c50013f5-4bb1-4b40-b62f-5b736b953ac0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:04.152793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:04.149848    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj" podUID="f4ff50fb-2482-4330-9ce4-e05d794c2a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:04.257919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:04.257572    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 20:10:04.257919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:04.257633    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" podUID="21980784-069f-41e3-ae48-40cd1c32186c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 20:10:04.446975 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:04.446915    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" podUID="87bb590b-e94c-4051-9df7-09be7673c1e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:04.523330 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:04.523225    5224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj" podUID="f4ff50fb-2482-4330-9ce4-e05d794c2a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 20:10:04.537115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:04.536995    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"
Nov 02 20:10:04.670709 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:04.670667    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6" podUID="f67432d1-22af-458b-bd57-002a02c62a5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:04.796093 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:04.794870    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:05.070982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.070872    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk" podUID="41edc495-0352-461a-a80c-c4dd7358e0c3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:05.074297 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.072665    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:10:05.075354 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.074664    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh" podUID="3359dc52-961f-43fd-b567-3bcb22271302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:05.100964 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.100914    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs" podUID="c50013f5-4bb1-4b40-b62f-5b736b953ac0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:05.101614 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.101593    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:10:05.254129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.237622    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6" podUID="675c9473-71e0-4bd1-8804-e684edf052fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 20:10:05.254708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.254572    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:10:05.254708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.254631    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" podUID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:05.311044 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.310907    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 20:10:05.311044 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.310973    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" podUID="21980784-069f-41e3-ae48-40cd1c32186c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 20:10:05.477857 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.462221    5224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6" podUID="f67432d1-22af-458b-bd57-002a02c62a5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded"
Nov 02 20:10:05.538794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.522890    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" podUID="87bb590b-e94c-4051-9df7-09be7673c1e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:05.538794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.523360    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:10:05.707808 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.689068    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6" podUID="f67432d1-22af-458b-bd57-002a02c62a5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:05.707808 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.689502    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:10:05.797797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.796052    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:05.797797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:05.796500    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:10:06.044591 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:06.044549    5224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" podUID="87bb590b-e94c-4051-9df7-09be7673c1e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Nov 02 20:10:06.093424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:06.093318    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk" podUID="41edc495-0352-461a-a80c-c4dd7358e0c3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 20:10:06.142932 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:06.142884    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj" podUID="f4ff50fb-2482-4330-9ce4-e05d794c2a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:06.172009 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:06.171966    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs" podUID="c50013f5-4bb1-4b40-b62f-5b736b953ac0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:06.246911 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:06.246862    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6" podUID="675c9473-71e0-4bd1-8804-e684edf052fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 20:10:06.247655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:06.247628    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:10:06.618804 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:06.617211    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" podUID="87bb590b-e94c-4051-9df7-09be7673c1e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 20:10:06.620125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:06.620094    5224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh" podUID="3359dc52-961f-43fd-b567-3bcb22271302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:06.691927 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:06.691875    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6" podUID="f67432d1-22af-458b-bd57-002a02c62a5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:06.803690 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:06.803636    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:07.071794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.068932    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh" podUID="3359dc52-961f-43fd-b567-3bcb22271302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 20:10:07.124880 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.123966    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk" podUID="41edc495-0352-461a-a80c-c4dd7358e0c3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:07.148798 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.146881    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj" podUID="f4ff50fb-2482-4330-9ce4-e05d794c2a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:07.148798 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.147702    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:10:07.176843 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.176799    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs" podUID="c50013f5-4bb1-4b40-b62f-5b736b953ac0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:07.288304 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.288272    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 20:10:07.292060 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.292021    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6" podUID="675c9473-71e0-4bd1-8804-e684edf052fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 20:10:07.336306 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.336210    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 20:10:07.382680 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.381570    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:10:07.385202 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.383701    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" podUID="21980784-069f-41e3-ae48-40cd1c32186c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:07.537422 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.537379    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 20:10:07.634401 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.632087    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:10:07.634401 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.633163    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 20:10:07.635898 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.635204    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" podUID="87bb590b-e94c-4051-9df7-09be7673c1e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:07.638494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.637641    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:10:07.652807 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.652007    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:10:07.652807 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.652459    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 20:10:07.677952 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.675848    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 20:10:07.821449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.821383    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:10:07.889738 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:07.889652    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 20:10:09.346339 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:09.346293    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:09.530175 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:09.530133    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:09.744654 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:09.744572    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:10.959494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:10.959445    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:15.904448 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:15.904404    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:17.156080 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:17.156043    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:20.792669 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:20.792622    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:20.992234 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:20.992190    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:21.167865 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.167786    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"]
Nov 02 20:10:21.170641 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.170609    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:21.173097 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.173075    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.263336 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.263299    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.263619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.263588    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.263745 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.263697    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.263920 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.263748    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.263920 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.263880    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.264047 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.263996    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvz5\" (UniqueName: \"kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-kube-api-access-7dvz5\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.264047 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.264024    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.264155 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.264077    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.264155 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.264126    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.280957 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.280923    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"]
Nov 02 20:10:21.365111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.365070    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.365300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.365124    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.365300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.365166    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.365300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.365207    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.365300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.365233    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.365300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.365263    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.365606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.365318    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvz5\" (UniqueName: \"kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-kube-api-access-7dvz5\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.365606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.365342    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.365606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.365391    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.365606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.365531    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.365829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.365703    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.365829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.365721    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.365949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.365879    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.366239 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.366214    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.367591 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.367564    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.367964 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.367942    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.423296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.423218    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.423424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.423332    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvz5\" (UniqueName: \"kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-kube-api-access-7dvz5\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.487307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.487268    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:21.909135 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:10:21.909099    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8f8a81_0c49_4ad7_be08_71a5e5bde231.slice/crio-7c6c3b4b7284034c338d1ec28e0b61ae4613fcca3a15220b6e056907f27b0816 WatchSource:0}: Error finding container 7c6c3b4b7284034c338d1ec28e0b61ae4613fcca3a15220b6e056907f27b0816: Status 404 returned error can't find the container with id 7c6c3b4b7284034c338d1ec28e0b61ae4613fcca3a15220b6e056907f27b0816
Nov 02 20:10:21.911619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.911586    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:10:21.911724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.911667    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:10:21.911724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.911694    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:10:21.913029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:21.913009    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"]
Nov 02 20:10:22.250502 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:22.250400    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf" event={"ID":"0d8f8a81-0c49-4ad7-be08-71a5e5bde231","Type":"ContainerStarted","Data":"9ab848341bf106b9b4e7927d87fce9b5a39f0f87fc7e2042760957ddaf593cc1"}
Nov 02 20:10:22.250502 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:22.250458    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf" event={"ID":"0d8f8a81-0c49-4ad7-be08-71a5e5bde231","Type":"ContainerStarted","Data":"7c6c3b4b7284034c338d1ec28e0b61ae4613fcca3a15220b6e056907f27b0816"}
Nov 02 20:10:22.369928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:22.369867    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf" podStartSLOduration=1.369848304 podStartE2EDuration="1.369848304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:22.3691157 +0000 UTC m=+1303.366804310" watchObservedRunningTime="2026-11-02 20:10:22.369848304 +0000 UTC m=+1303.367536913"
Nov 02 20:10:22.488195 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:22.488156    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:22.489942 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:22.489914    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 20:10:22.490051 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:22.489971    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf" podUID="0d8f8a81-0c49-4ad7-be08-71a5e5bde231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 20:10:23.488259 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:23.488216    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 20:10:23.488682 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:23.488275    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf" podUID="0d8f8a81-0c49-4ad7-be08-71a5e5bde231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 20:10:24.488574 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:24.488531    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 20:10:24.489020 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:24.488601    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf" podUID="0d8f8a81-0c49-4ad7-be08-71a5e5bde231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 20:10:25.488085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:25.488036    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 20:10:25.488274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:25.488130    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf" podUID="0d8f8a81-0c49-4ad7-be08-71a5e5bde231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 20:10:25.883901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:25.883861    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:26.311215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:26.311163    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:26.492792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:26.492139    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 20:10:26.492792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:26.492215    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf" podUID="0d8f8a81-0c49-4ad7-be08-71a5e5bde231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 20:10:27.492793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:27.492546    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 20:10:27.492793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:27.492640    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf" podUID="0d8f8a81-0c49-4ad7-be08-71a5e5bde231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 20:10:27.540557 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:10:27.540503    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:10:28.487993 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:28.487958    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 20:10:28.488172 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:28.488013    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf" podUID="0d8f8a81-0c49-4ad7-be08-71a5e5bde231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 20:10:29.639501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:29.639451    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:29.640024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:29.639878    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:29.640828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:29.640806    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:10:31.473144 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:31.473105    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:33.809609 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:33.809566    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:34.257459 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.257372    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"]
Nov 02 20:10:34.261676 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.261656    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.319731 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.319689    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"]
Nov 02 20:10:34.320447 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.320412    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-certs\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.320566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.320477    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-credential-socket\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.320566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.320524    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-istio-token\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.320566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.320550    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf37ca7-a400-495d-bce4-cf6613da5355-istiod-ca-cert\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.320720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.320595    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25tcm\" (UniqueName: \"kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-kube-api-access-25tcm\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.320720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.320666    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-data\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.320858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.320721    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-envoy\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.320858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.320751    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf37ca7-a400-495d-bce4-cf6613da5355-istio-podinfo\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.320858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.320804    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-socket\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.363798 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.363750    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-45c182-openshift-default-dockercfg-z6vsk\""
Nov 02 20:10:34.385881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.385848    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:34.422496 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.422458    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-istio-token\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.422496 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.422502    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf37ca7-a400-495d-bce4-cf6613da5355-istiod-ca-cert\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.422763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.422540    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25tcm\" (UniqueName: \"kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-kube-api-access-25tcm\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.422763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.422600    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-data\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.422763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.422645    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-envoy\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.422763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.422675    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf37ca7-a400-495d-bce4-cf6613da5355-istio-podinfo\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.422763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.422719    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-socket\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.423034 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.422926    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-certs\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.423034 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.422974    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-credential-socket\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.423413 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.423385    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-credential-socket\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.424493 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.424451    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf37ca7-a400-495d-bce4-cf6613da5355-istiod-ca-cert\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.424992 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.424971    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-data\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.425481 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.425458    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-socket\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.425897 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.425874    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-certs\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.427928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.427876    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-envoy\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.428513 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.428489    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf37ca7-a400-495d-bce4-cf6613da5355-istio-podinfo\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.492396 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.492362    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-istio-token\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.497048 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.496972    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25tcm\" (UniqueName: \"kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-kube-api-access-25tcm\") pod \"e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") " pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.577551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.577185    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:34.768184 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:34.768140    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:35.025343 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:10:35.025311    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf37ca7_a400_495d_bce4_cf6613da5355.slice/crio-691f873bbe2af65f4054292fb5d1dd45d7a5a3df82c5f20d8dc3db8414567a0f WatchSource:0}: Error finding container 691f873bbe2af65f4054292fb5d1dd45d7a5a3df82c5f20d8dc3db8414567a0f: Status 404 returned error can't find the container with id 691f873bbe2af65f4054292fb5d1dd45d7a5a3df82c5f20d8dc3db8414567a0f
Nov 02 20:10:35.028344 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:35.028308    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:10:35.028442 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:35.028378    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:10:35.028442 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:35.028409    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:10:35.030310 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:35.030280    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"]
Nov 02 20:10:35.481107 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:35.481017    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" event={"ID":"baf37ca7-a400-495d-bce4-cf6613da5355","Type":"ContainerStarted","Data":"db602aede4931e47a7f47abb2c97bb5bf89b56232a5521a9c5290fd681d328bb"}
Nov 02 20:10:35.481107 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:35.481077    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" event={"ID":"baf37ca7-a400-495d-bce4-cf6613da5355","Type":"ContainerStarted","Data":"691f873bbe2af65f4054292fb5d1dd45d7a5a3df82c5f20d8dc3db8414567a0f"}
Nov 02 20:10:35.578135 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:35.578095    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:35.600648 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:35.600580    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" podStartSLOduration=1.600565324 podStartE2EDuration="1.600565324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:35.59835905 +0000 UTC m=+1316.596047659" watchObservedRunningTime="2026-11-02 20:10:35.600565324 +0000 UTC m=+1316.598253933"
Nov 02 20:10:36.367682 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.367643    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:36.526853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.526816    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"]
Nov 02 20:10:36.530864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.530847    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"
Nov 02 20:10:36.585162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.585128    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:10:36.585381 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.585217    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:10:36.585381 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.585298    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:10:36.585463 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.585408    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:10:36.585986 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.585952    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:36.596855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.596821    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"]
Nov 02 20:10:36.648822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.648693    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa5b11e8-t97kr\" (UID: \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"
Nov 02 20:10:36.648822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.648753    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa5b11e8-t97kr\" (UID: \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"
Nov 02 20:10:36.649028 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.648954    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rsk\" (UniqueName: \"kubernetes.io/projected/18cc3e9f-b233-4e43-a40a-5e50bec1f680-kube-api-access-76rsk\") pod \"maas-api-revoke-sub-aa5b11e8-t97kr\" (UID: \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"
Nov 02 20:10:36.750718 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.750682    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rsk\" (UniqueName: \"kubernetes.io/projected/18cc3e9f-b233-4e43-a40a-5e50bec1f680-kube-api-access-76rsk\") pod \"maas-api-revoke-sub-aa5b11e8-t97kr\" (UID: \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"
Nov 02 20:10:36.750968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.750817    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa5b11e8-t97kr\" (UID: \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"
Nov 02 20:10:36.750968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.750865    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa5b11e8-t97kr\" (UID: \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"
Nov 02 20:10:36.751529 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.751507    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa5b11e8-t97kr\" (UID: \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"
Nov 02 20:10:36.753333 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.753312    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa5b11e8-t97kr\" (UID: \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"
Nov 02 20:10:36.809779 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.809735    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rsk\" (UniqueName: \"kubernetes.io/projected/18cc3e9f-b233-4e43-a40a-5e50bec1f680-kube-api-access-76rsk\") pod \"maas-api-revoke-sub-aa5b11e8-t97kr\" (UID: \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"
Nov 02 20:10:36.860666 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:36.860619    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"
Nov 02 20:10:37.234260 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:10:37.234224    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cc3e9f_b233_4e43_a40a_5e50bec1f680.slice/crio-b7c8efd35167540b1b65900bd2570067c6f2d0bb174611601005735eb7a69b14 WatchSource:0}: Error finding container b7c8efd35167540b1b65900bd2570067c6f2d0bb174611601005735eb7a69b14: Status 404 returned error can't find the container with id b7c8efd35167540b1b65900bd2570067c6f2d0bb174611601005735eb7a69b14
Nov 02 20:10:37.238500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:37.238424    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"]
Nov 02 20:10:37.493672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:37.493583    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr" event={"ID":"18cc3e9f-b233-4e43-a40a-5e50bec1f680","Type":"ContainerStarted","Data":"b7c8efd35167540b1b65900bd2570067c6f2d0bb174611601005735eb7a69b14"}
Nov 02 20:10:37.494157 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:37.493804    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:37.494885 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:37.494865    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:10:38.433023 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:38.432931    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:38.501416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:38.501377    5224 generic.go:356] "Generic (PLEG): container finished" podID="18cc3e9f-b233-4e43-a40a-5e50bec1f680" containerID="5d9cc1133b38a2b821f00a558d1c45a0959adfa665c010cc8924682303dc2577" exitCode=0
Nov 02 20:10:38.501905 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:38.501466    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr" event={"ID":"18cc3e9f-b233-4e43-a40a-5e50bec1f680","Type":"ContainerDied","Data":"5d9cc1133b38a2b821f00a558d1c45a0959adfa665c010cc8924682303dc2577"}
Nov 02 20:10:38.615574 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:38.615536    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:38.799100 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:38.799064    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:39.471925 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:39.471892    5224 scope.go:122] "RemoveContainer" containerID="46b0dcba66bfc488ada20b9ef6d34514940f91e94192b2038e30f5fe1bad77fd"
Nov 02 20:10:39.668077 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:39.668047    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"
Nov 02 20:10:39.797046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:39.797011    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18cc3e9f-b233-4e43-a40a-5e50bec1f680-kube-api-access-76rsk\" (UniqueName: \"kubernetes.io/projected/18cc3e9f-b233-4e43-a40a-5e50bec1f680-kube-api-access-76rsk\") pod \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\" (UID: \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\") "
Nov 02 20:10:39.797219 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:39.797194    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-xks-ca\") pod \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\" (UID: \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\") "
Nov 02 20:10:39.797302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:39.797288    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-openshift-ca\") pod \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\" (UID: \"18cc3e9f-b233-4e43-a40a-5e50bec1f680\") "
Nov 02 20:10:39.797728 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:39.797695    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-openshift-ca" pod "18cc3e9f-b233-4e43-a40a-5e50bec1f680" (UID: "18cc3e9f-b233-4e43-a40a-5e50bec1f680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:10:39.799377 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:39.799351    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cc3e9f-b233-4e43-a40a-5e50bec1f680-kube-api-access-76rsk" pod "18cc3e9f-b233-4e43-a40a-5e50bec1f680" (UID: "18cc3e9f-b233-4e43-a40a-5e50bec1f680"). InnerVolumeSpecName "kube-api-access-76rsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:10:39.799701 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:39.799683    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-xks-ca" pod "18cc3e9f-b233-4e43-a40a-5e50bec1f680" (UID: "18cc3e9f-b233-4e43-a40a-5e50bec1f680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:10:39.898699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:39.898655    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:39.898699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:39.898699    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc3e9f-b233-4e43-a40a-5e50bec1f680-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:39.898950 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:39.898716    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76rsk\" (UniqueName: \"kubernetes.io/projected/18cc3e9f-b233-4e43-a40a-5e50bec1f680-kube-api-access-76rsk\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:40.184996 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:40.184913    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:10:40.515394 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:40.515356    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr" event={"ID":"18cc3e9f-b233-4e43-a40a-5e50bec1f680","Type":"ContainerDied","Data":"b7c8efd35167540b1b65900bd2570067c6f2d0bb174611601005735eb7a69b14"}
Nov 02 20:10:40.515564 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:40.515413    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7c8efd35167540b1b65900bd2570067c6f2d0bb174611601005735eb7a69b14"
Nov 02 20:10:40.515564 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:40.515375    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"
Nov 02 20:10:41.631315 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:41.631271    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:41.829705 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:41.829667    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:41.983178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:41.983098    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"]
Nov 02 20:10:41.984072 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:41.984051    5224 cpu_manager.go:414] "containerMap: removing container" podUID="18cc3e9f-b233-4e43-a40a-5e50bec1f680" containerName="revoke-keys"
Nov 02 20:10:41.984159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:41.984076    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cc3e9f-b233-4e43-a40a-5e50bec1f680" containerName="revoke-keys"
Nov 02 20:10:41.984226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:41.984214    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cc3e9f-b233-4e43-a40a-5e50bec1f680" containerName="revoke-keys"
Nov 02 20:10:41.987618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:41.987598    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"
Nov 02 20:10:42.006158 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.006118    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:42.037713 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.037675    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"]
Nov 02 20:10:42.091189 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.091153    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-45c182\""
Nov 02 20:10:42.091632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.091610    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-45c182-dockercfg-cz7rr\""
Nov 02 20:10:42.139973 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.139914    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jrgt\" (UniqueName: \"kubernetes.io/projected/d26c20ab-f400-439e-8f58-883b3fe49d2a-kube-api-access-8jrgt\") pod \"payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97\" (UID: \"d26c20ab-f400-439e-8f58-883b3fe49d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"
Nov 02 20:10:42.140416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.140385    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26c20ab-f400-439e-8f58-883b3fe49d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97\" (UID: \"d26c20ab-f400-439e-8f58-883b3fe49d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"
Nov 02 20:10:42.170792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.169592    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"]
Nov 02 20:10:42.186374 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.186335    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"
Nov 02 20:10:42.206226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.206193    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"]
Nov 02 20:10:42.241533 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.241423    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jrgt\" (UniqueName: \"kubernetes.io/projected/d26c20ab-f400-439e-8f58-883b3fe49d2a-kube-api-access-8jrgt\") pod \"payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97\" (UID: \"d26c20ab-f400-439e-8f58-883b3fe49d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"
Nov 02 20:10:42.241721 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.241536    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26c20ab-f400-439e-8f58-883b3fe49d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97\" (UID: \"d26c20ab-f400-439e-8f58-883b3fe49d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"
Nov 02 20:10:42.242602 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.242573    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26c20ab-f400-439e-8f58-883b3fe49d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97\" (UID: \"d26c20ab-f400-439e-8f58-883b3fe49d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"
Nov 02 20:10:42.299366 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.299330    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jrgt\" (UniqueName: \"kubernetes.io/projected/d26c20ab-f400-439e-8f58-883b3fe49d2a-kube-api-access-8jrgt\") pod \"payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97\" (UID: \"d26c20ab-f400-439e-8f58-883b3fe49d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"
Nov 02 20:10:42.306270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.306234    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"
Nov 02 20:10:42.342460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.342419    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5v5n\" (UniqueName: \"kubernetes.io/projected/17665ced-46f5-4128-bcf5-0779a5562797-kube-api-access-z5v5n\") pod \"payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc\" (UID: \"17665ced-46f5-4128-bcf5-0779a5562797\") " pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"
Nov 02 20:10:42.342641 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.342498    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17665ced-46f5-4128-bcf5-0779a5562797-plugins-config-volume\") pod \"payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc\" (UID: \"17665ced-46f5-4128-bcf5-0779a5562797\") " pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"
Nov 02 20:10:42.443863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.443819    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5v5n\" (UniqueName: \"kubernetes.io/projected/17665ced-46f5-4128-bcf5-0779a5562797-kube-api-access-z5v5n\") pod \"payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc\" (UID: \"17665ced-46f5-4128-bcf5-0779a5562797\") " pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"
Nov 02 20:10:42.444050 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.443889    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17665ced-46f5-4128-bcf5-0779a5562797-plugins-config-volume\") pod \"payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc\" (UID: \"17665ced-46f5-4128-bcf5-0779a5562797\") " pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"
Nov 02 20:10:42.444607 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.444586    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17665ced-46f5-4128-bcf5-0779a5562797-plugins-config-volume\") pod \"payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc\" (UID: \"17665ced-46f5-4128-bcf5-0779a5562797\") " pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"
Nov 02 20:10:42.501842 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.501730    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5v5n\" (UniqueName: \"kubernetes.io/projected/17665ced-46f5-4128-bcf5-0779a5562797-kube-api-access-z5v5n\") pod \"payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc\" (UID: \"17665ced-46f5-4128-bcf5-0779a5562797\") " pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"
Nov 02 20:10:42.506450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.506426    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"
Nov 02 20:10:42.682004 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:10:42.681968    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26c20ab_f400_439e_8f58_883b3fe49d2a.slice/crio-8a39676d84135e7ef656ffb2e883cefcbd72e23f4f6d7dd1f469d03bd7bd8d30 WatchSource:0}: Error finding container 8a39676d84135e7ef656ffb2e883cefcbd72e23f4f6d7dd1f469d03bd7bd8d30: Status 404 returned error can't find the container with id 8a39676d84135e7ef656ffb2e883cefcbd72e23f4f6d7dd1f469d03bd7bd8d30
Nov 02 20:10:42.685257 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.685233    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"]
Nov 02 20:10:42.873558 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:10:42.873514    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17665ced_46f5_4128_bcf5_0779a5562797.slice/crio-34d0d6fb2169f70b1b20e8ece024d51731fccc7df3a5b8021302d632147334db WatchSource:0}: Error finding container 34d0d6fb2169f70b1b20e8ece024d51731fccc7df3a5b8021302d632147334db: Status 404 returned error can't find the container with id 34d0d6fb2169f70b1b20e8ece024d51731fccc7df3a5b8021302d632147334db
Nov 02 20:10:42.877654 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:42.877625    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"]
Nov 02 20:10:43.539162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:43.539126    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97" event={"ID":"d26c20ab-f400-439e-8f58-883b3fe49d2a","Type":"ContainerStarted","Data":"150e7b09bad7e0bb22d21a438998be41c6677203ae4de090e9f983fb07be6dfa"}
Nov 02 20:10:43.539366 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:43.539199    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97" event={"ID":"d26c20ab-f400-439e-8f58-883b3fe49d2a","Type":"ContainerStarted","Data":"8a39676d84135e7ef656ffb2e883cefcbd72e23f4f6d7dd1f469d03bd7bd8d30"}
Nov 02 20:10:43.539366 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:43.539332    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"
Nov 02 20:10:43.540741 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:43.540718    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc" event={"ID":"17665ced-46f5-4128-bcf5-0779a5562797","Type":"ContainerStarted","Data":"e98a881dc2f7ffed2255f4d3b3e1775cf83344b73d71be70fe7269d18e2e40ba"}
Nov 02 20:10:43.540890 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:43.540756    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc" event={"ID":"17665ced-46f5-4128-bcf5-0779a5562797","Type":"ContainerStarted","Data":"34d0d6fb2169f70b1b20e8ece024d51731fccc7df3a5b8021302d632147334db"}
Nov 02 20:10:43.540890 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:43.540810    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"
Nov 02 20:10:43.656587 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:43.656532    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97" podStartSLOduration=2.656516073 podStartE2EDuration="2.656516073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:43.65407761 +0000 UTC m=+1324.651766221" watchObservedRunningTime="2026-11-02 20:10:43.656516073 +0000 UTC m=+1324.654204682"
Nov 02 20:10:43.771994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:43.771940    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc" podStartSLOduration=1.771925865 podStartE2EDuration="1.771925865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:43.769257138 +0000 UTC m=+1324.766945758" watchObservedRunningTime="2026-11-02 20:10:43.771925865 +0000 UTC m=+1324.769614474"
Nov 02 20:10:47.243801 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:47.243312    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:47.682165 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:47.682129    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:48.338590 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:48.338356    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:48.684238 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:48.684138    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:52.080379 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:52.080336    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:52.279048 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:52.279006    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:52.675762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:52.675716    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:52.863882 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:52.863845    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:54.050002 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:54.049963    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:54.558032 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:54.558000    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"
Nov 02 20:10:54.558225 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:54.558172    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"
Nov 02 20:10:55.845455 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:55.845412    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:56.064404 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.064371    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:56.207818 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.207706    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"]
Nov 02 20:10:56.211619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.211593    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"
Nov 02 20:10:56.264183 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.264147    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"]
Nov 02 20:10:56.319139 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.319106    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-08eeaa\""
Nov 02 20:10:56.319396 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.319113    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-08eeaa-dockercfg-96bvp\""
Nov 02 20:10:56.337047 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.336996    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67qzv\" (UniqueName: \"kubernetes.io/projected/73c7a129-8af4-41d3-bcd5-84c070f1843e-kube-api-access-67qzv\") pod \"payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx\" (UID: \"73c7a129-8af4-41d3-bcd5-84c070f1843e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"
Nov 02 20:10:56.337436 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.337411    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c7a129-8af4-41d3-bcd5-84c070f1843e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx\" (UID: \"73c7a129-8af4-41d3-bcd5-84c070f1843e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"
Nov 02 20:10:56.371418 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.371386    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"]
Nov 02 20:10:56.397179 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.397140    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"
Nov 02 20:10:56.432510 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.432464    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"]
Nov 02 20:10:56.444823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.441614    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c7a129-8af4-41d3-bcd5-84c070f1843e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx\" (UID: \"73c7a129-8af4-41d3-bcd5-84c070f1843e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"
Nov 02 20:10:56.444823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.441675    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qzv\" (UniqueName: \"kubernetes.io/projected/73c7a129-8af4-41d3-bcd5-84c070f1843e-kube-api-access-67qzv\") pod \"payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx\" (UID: \"73c7a129-8af4-41d3-bcd5-84c070f1843e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"
Nov 02 20:10:56.444823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.443140    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c7a129-8af4-41d3-bcd5-84c070f1843e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx\" (UID: \"73c7a129-8af4-41d3-bcd5-84c070f1843e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"
Nov 02 20:10:56.499728 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.499655    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qzv\" (UniqueName: \"kubernetes.io/projected/73c7a129-8af4-41d3-bcd5-84c070f1843e-kube-api-access-67qzv\") pod \"payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx\" (UID: \"73c7a129-8af4-41d3-bcd5-84c070f1843e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"
Nov 02 20:10:56.542680 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.542644    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc4e74f-60ce-4443-a161-505979d549e7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx\" (UID: \"3bc4e74f-60ce-4443-a161-505979d549e7\") " pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"
Nov 02 20:10:56.542886 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.542707    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm8r5\" (UniqueName: \"kubernetes.io/projected/3bc4e74f-60ce-4443-a161-505979d549e7-kube-api-access-mm8r5\") pod \"payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx\" (UID: \"3bc4e74f-60ce-4443-a161-505979d549e7\") " pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"
Nov 02 20:10:56.551179 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.551154    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"
Nov 02 20:10:56.643578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.643541    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc4e74f-60ce-4443-a161-505979d549e7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx\" (UID: \"3bc4e74f-60ce-4443-a161-505979d549e7\") " pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"
Nov 02 20:10:56.643787 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.643609    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8r5\" (UniqueName: \"kubernetes.io/projected/3bc4e74f-60ce-4443-a161-505979d549e7-kube-api-access-mm8r5\") pod \"payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx\" (UID: \"3bc4e74f-60ce-4443-a161-505979d549e7\") " pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"
Nov 02 20:10:56.644356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.644332    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc4e74f-60ce-4443-a161-505979d549e7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx\" (UID: \"3bc4e74f-60ce-4443-a161-505979d549e7\") " pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"
Nov 02 20:10:56.701615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.701583    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8r5\" (UniqueName: \"kubernetes.io/projected/3bc4e74f-60ce-4443-a161-505979d549e7-kube-api-access-mm8r5\") pod \"payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx\" (UID: \"3bc4e74f-60ce-4443-a161-505979d549e7\") " pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"
Nov 02 20:10:56.749827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.749702    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"
Nov 02 20:10:56.917940 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:10:56.917902    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c7a129_8af4_41d3_bcd5_84c070f1843e.slice/crio-4176db5a2d48c6b701e19700491bcc89ad0582b10619a150c01a1ea36aace536 WatchSource:0}: Error finding container 4176db5a2d48c6b701e19700491bcc89ad0582b10619a150c01a1ea36aace536: Status 404 returned error can't find the container with id 4176db5a2d48c6b701e19700491bcc89ad0582b10619a150c01a1ea36aace536
Nov 02 20:10:56.921826 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:56.921800    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"]
Nov 02 20:10:57.118058 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:10:57.118030    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc4e74f_60ce_4443_a161_505979d549e7.slice/crio-89cd8d70661b52e56100f45a9114f0b9fa258d0c029255b7155e3589683abe01 WatchSource:0}: Error finding container 89cd8d70661b52e56100f45a9114f0b9fa258d0c029255b7155e3589683abe01: Status 404 returned error can't find the container with id 89cd8d70661b52e56100f45a9114f0b9fa258d0c029255b7155e3589683abe01
Nov 02 20:10:57.121602 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:57.121579    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"]
Nov 02 20:10:57.359640 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:10:57.359606    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:10:57.447227 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:57.447121    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:10:57.644188 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:57.644145    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx" event={"ID":"3bc4e74f-60ce-4443-a161-505979d549e7","Type":"ContainerStarted","Data":"1fa8ff778602471ab961dbd8f184e663d323ec1f5d3c0a4efecf95ad06d75e0f"}
Nov 02 20:10:57.644367 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:57.644218    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"
Nov 02 20:10:57.644367 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:57.644242    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx" event={"ID":"3bc4e74f-60ce-4443-a161-505979d549e7","Type":"ContainerStarted","Data":"89cd8d70661b52e56100f45a9114f0b9fa258d0c029255b7155e3589683abe01"}
Nov 02 20:10:57.645739 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:57.645715    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx" event={"ID":"73c7a129-8af4-41d3-bcd5-84c070f1843e","Type":"ContainerStarted","Data":"98bf424aadc7f54aede80818285a0f60b65d7e6c9901744b4a2cc126a2808e5e"}
Nov 02 20:10:57.645899 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:57.645749    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx" event={"ID":"73c7a129-8af4-41d3-bcd5-84c070f1843e","Type":"ContainerStarted","Data":"4176db5a2d48c6b701e19700491bcc89ad0582b10619a150c01a1ea36aace536"}
Nov 02 20:10:57.645899 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:57.645840    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"
Nov 02 20:10:57.761169 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:57.761072    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx" podStartSLOduration=1.76105847 podStartE2EDuration="1.76105847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:57.758082031 +0000 UTC m=+1338.755770639" watchObservedRunningTime="2026-11-02 20:10:57.76105847 +0000 UTC m=+1338.758747119"
Nov 02 20:10:57.873967 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:10:57.873907    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx" podStartSLOduration=1.873889653 podStartE2EDuration="1.873889653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:57.872424278 +0000 UTC m=+1338.870112887" watchObservedRunningTime="2026-11-02 20:10:57.873889653 +0000 UTC m=+1338.871578264"
Nov 02 20:11:02.568280 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:02.568229    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:05.084431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:05.084380    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:08.250477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.250442    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:08.655629 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.655597    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"
Nov 02 20:11:08.655858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.655680    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"
Nov 02 20:11:08.764164 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.764126    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"]
Nov 02 20:11:08.768719 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.768685    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.795938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.795905    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"]
Nov 02 20:11:08.808149 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.808106    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sftm5\" (UniqueName: \"kubernetes.io/projected/c2b4bca4-928d-439e-b582-979ff1768fc6-kube-api-access-sftm5\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.808357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.808196    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-model-cache\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.808515 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.808485    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-home\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.808606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.808565    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-dshm\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.808606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.808595    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b4bca4-928d-439e-b582-979ff1768fc6-tls-certs\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.808722 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.808639    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-tmp-dir\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.824863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.824832    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6a4ab0\"/\"distinct-w3-6a4ab0-kserve-self-signed-certs\""
Nov 02 20:11:08.876476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.876436    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:08.910070 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.909974    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-home\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.910376 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.910352    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-dshm\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.910514 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.910499    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b4bca4-928d-439e-b582-979ff1768fc6-tls-certs\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.910684 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.910658    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-tmp-dir\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.910892 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.910874    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sftm5\" (UniqueName: \"kubernetes.io/projected/c2b4bca4-928d-439e-b582-979ff1768fc6-kube-api-access-sftm5\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.911070 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.911053    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-model-cache\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.912425 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.912398    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-model-cache\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.912710 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.912690    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-home\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.913352 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.913326    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-tmp-dir\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.914884 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.914839    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-dshm\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.916108 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.916077    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b4bca4-928d-439e-b582-979ff1768fc6-tls-certs\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:08.971529 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:08.971495    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sftm5\" (UniqueName: \"kubernetes.io/projected/c2b4bca4-928d-439e-b582-979ff1768fc6-kube-api-access-sftm5\") pod \"distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:09.090674 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:09.090617    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:09.542694 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:11:09.542659    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2b4bca4_928d_439e_b582_979ff1768fc6.slice/crio-99ca4c2b73e8a032e35ea680d56cf1943ce869774c6c73c4f3ef8c49b64f7f9e WatchSource:0}: Error finding container 99ca4c2b73e8a032e35ea680d56cf1943ce869774c6c73c4f3ef8c49b64f7f9e: Status 404 returned error can't find the container with id 99ca4c2b73e8a032e35ea680d56cf1943ce869774c6c73c4f3ef8c49b64f7f9e
Nov 02 20:11:09.547298 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:09.547272    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"]
Nov 02 20:11:09.754749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:09.754716    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg" event={"ID":"c2b4bca4-928d-439e-b582-979ff1768fc6","Type":"ContainerStarted","Data":"2235f79493f9f3c820b51149a6688b716eae5dd107c19c9ffa85404c9b0482cf"}
Nov 02 20:11:09.754960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:09.754758    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg" event={"ID":"c2b4bca4-928d-439e-b582-979ff1768fc6","Type":"ContainerStarted","Data":"99ca4c2b73e8a032e35ea680d56cf1943ce869774c6c73c4f3ef8c49b64f7f9e"}
Nov 02 20:11:09.875851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:09.873794    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg" podStartSLOduration=1.873759868 podStartE2EDuration="1.873759868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:09.870026091 +0000 UTC m=+1350.867714700" watchObservedRunningTime="2026-11-02 20:11:09.873759868 +0000 UTC m=+1350.871448481"
Nov 02 20:11:10.090902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:10.090860    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:13.180603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:13.180560    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:17.889787 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:17.889728    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:18.064565 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:18.064483    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:18.238635 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:18.238543    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:19.091470 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:19.091435    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:19.104800 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:19.104757    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:19.840381 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:19.840336    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 20:11:22.949931 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:22.949885    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:26.659714 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.659672    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"]
Nov 02 20:11:26.663760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.663733    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.723562 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.723531    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"]
Nov 02 20:11:26.761776 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.761729    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-model-cache\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.761948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.761786    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-tmp-dir\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.761948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.761898    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tdpb\" (UniqueName: \"kubernetes.io/projected/835fa46e-2a18-4009-a5a4-51e3599c161a-kube-api-access-8tdpb\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.762031 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.762016    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-dshm\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.762081 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.762068    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-home\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.762212 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.762199    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835fa46e-2a18-4009-a5a4-51e3599c161a-tls-certs\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.771161 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.771133    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5e04421b\"/\"e2e-lifecycle-model-5e04421b-kserve-self-signed-certs\""
Nov 02 20:11:26.771426 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.771378    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5e04421b\"/\"default-dockercfg-vhhs2\""
Nov 02 20:11:26.773840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.773809    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5e04421b\"/\"openshift-service-ca.crt\""
Nov 02 20:11:26.773974 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.773859    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5e04421b\"/\"kube-root-ca.crt\""
Nov 02 20:11:26.777939 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.777898    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:26.864246 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.864208    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-dshm\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.864415 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.864285    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-home\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.864415 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.864401    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835fa46e-2a18-4009-a5a4-51e3599c161a-tls-certs\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.864488 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.864472    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-model-cache\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.864530 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.864508    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-tmp-dir\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.864573 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.864559    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tdpb\" (UniqueName: \"kubernetes.io/projected/835fa46e-2a18-4009-a5a4-51e3599c161a-kube-api-access-8tdpb\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.865431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.865401    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-home\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.865584 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.865467    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-model-cache\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.865750 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.865727    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-tmp-dir\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.867349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.867319    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-dshm\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.867499 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.867484    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835fa46e-2a18-4009-a5a4-51e3599c161a-tls-certs\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.924843 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.924732    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tdpb\" (UniqueName: \"kubernetes.io/projected/835fa46e-2a18-4009-a5a4-51e3599c161a-kube-api-access-8tdpb\") pod \"e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") " pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:26.981225 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:26.981179    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:27.357057 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:11:27.357025    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod835fa46e_2a18_4009_a5a4_51e3599c161a.slice/crio-672883f5b62a67f1109aa996ad3159593f5a2652c4d20bd9df60543a37f03d28 WatchSource:0}: Error finding container 672883f5b62a67f1109aa996ad3159593f5a2652c4d20bd9df60543a37f03d28: Status 404 returned error can't find the container with id 672883f5b62a67f1109aa996ad3159593f5a2652c4d20bd9df60543a37f03d28
Nov 02 20:11:27.361756 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:11:27.361723    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:11:27.362078 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:27.362055    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"]
Nov 02 20:11:27.890346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:27.890308    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs" event={"ID":"835fa46e-2a18-4009-a5a4-51e3599c161a","Type":"ContainerStarted","Data":"1ea832f86a1043f3cb559c9113394e3feb9bad661bc372b1a5a589495830af09"}
Nov 02 20:11:27.890763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:27.890358    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs" event={"ID":"835fa46e-2a18-4009-a5a4-51e3599c161a","Type":"ContainerStarted","Data":"672883f5b62a67f1109aa996ad3159593f5a2652c4d20bd9df60543a37f03d28"}
Nov 02 20:11:27.982067 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:27.982011    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:28.013624 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:28.013573    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs" podStartSLOduration=2.013558922 podStartE2EDuration="2.013558922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:28.013032238 +0000 UTC m=+1369.010720859" watchObservedRunningTime="2026-11-02 20:11:28.013558922 +0000 UTC m=+1369.011247531"
Nov 02 20:11:28.021219 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:28.021186    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:28.079619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:28.079582    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:11:32.799620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:32.799573    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:33.019285 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:33.019243    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:33.449485 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:33.449444    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:36.981934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:36.981897    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:36.994685 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:36.994658    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:37.950458 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:37.950414    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:37.978166 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:37.978139    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:11:42.852006 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:42.851967    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:47.775080 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:47.775042    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:47.980079 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:47.980042    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:48.583365 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:48.583318    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:48.975259 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:48.975164    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:49.754670 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:49.754633    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:11:50.436424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:50.436391    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:52.408936 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:52.408897    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:52.982658 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:52.982617    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:53.329024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.328992    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:53.506432 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.506392    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:53.711582 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.711490    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"]
Nov 02 20:11:53.716563 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.716539    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.778569 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.778536    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6a4ab0-openshift-default-dockercfg-xzj7t\""
Nov 02 20:11:53.793799 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.793735    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:53.828855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.828822    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"]
Nov 02 20:11:53.871557 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.871516    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbtvt\" (UniqueName: \"kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-kube-api-access-mbtvt\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.873943 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.873912    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.874089 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.874013    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.874156 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.874103    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.874226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.874207    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.874287 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.874233    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.874352 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.874339    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.874396 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.874369    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.874432 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.874396    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f35ea3d-b5b9-47b2-800d-78cc466867af-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.975578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.975485    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.975578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.975530    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.975823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.975583    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.975823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.975676    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.975823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.975700    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.975823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.975821    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.976034 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.975854    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.976034 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.975882    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f35ea3d-b5b9-47b2-800d-78cc466867af-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.976034 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.975946    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.976034 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.975965    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbtvt\" (UniqueName: \"kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-kube-api-access-mbtvt\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.976259 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.976176    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.976703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.976652    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.976703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.976688    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.976926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.976902    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f35ea3d-b5b9-47b2-800d-78cc466867af-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.978891 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.978870    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:53.979050 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:53.979031    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:54.033844 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:54.033821    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:54.033962 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:54.033932    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbtvt\" (UniqueName: \"kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-kube-api-access-mbtvt\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:54.330175 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:54.330143    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:54.777808 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:54.777747    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"]
Nov 02 20:11:54.778509 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:11:54.778479    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f35ea3d_b5b9_47b2_800d_78cc466867af.slice/crio-c1e0cf610dc73c17e4aec414d43babe9700124d0c10266305d17286de268570f WatchSource:0}: Error finding container c1e0cf610dc73c17e4aec414d43babe9700124d0c10266305d17286de268570f: Status 404 returned error can't find the container with id c1e0cf610dc73c17e4aec414d43babe9700124d0c10266305d17286de268570f
Nov 02 20:11:54.781094 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:54.781062    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:11:54.781178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:54.781129    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:11:54.781178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:54.781157    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:11:55.103808 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:55.103690    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" event={"ID":"9f35ea3d-b5b9-47b2-800d-78cc466867af","Type":"ContainerStarted","Data":"5cf2aa375b80445714c151924b8c4b658ba40ff4006dd8036cd44f3bc4e1b167"}
Nov 02 20:11:55.103948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:55.103813    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" event={"ID":"9f35ea3d-b5b9-47b2-800d-78cc466867af","Type":"ContainerStarted","Data":"c1e0cf610dc73c17e4aec414d43babe9700124d0c10266305d17286de268570f"}
Nov 02 20:11:55.230489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:55.230432    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" podStartSLOduration=2.2304165830000002 podStartE2EDuration="2.230416583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:55.227253822 +0000 UTC m=+1396.224942433" watchObservedRunningTime="2026-11-02 20:11:55.230416583 +0000 UTC m=+1396.228105183"
Nov 02 20:11:55.331265 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:55.330867    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:11:55.332306 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:55.332275    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 20:11:55.332444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:55.332331    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" podUID="9f35ea3d-b5b9-47b2-800d-78cc466867af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 20:11:56.331496 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:56.331460    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 20:11:56.331923 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:56.331519    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" podUID="9f35ea3d-b5b9-47b2-800d-78cc466867af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 20:11:56.574928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:56.574884    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:56.791132 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:56.791035    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:57.330461 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:57.330418    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 20:11:57.330660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:57.330475    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" podUID="9f35ea3d-b5b9-47b2-800d-78cc466867af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 20:11:57.381579 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:11:57.381534    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:11:58.331480 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:58.331441    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 20:11:58.331694 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:58.331506    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" podUID="9f35ea3d-b5b9-47b2-800d-78cc466867af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 20:11:58.698205 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:58.698121    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:11:59.330548 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:59.330512    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 20:11:59.330723 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:11:59.330576    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" podUID="9f35ea3d-b5b9-47b2-800d-78cc466867af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 20:12:00.331209 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:00.331175    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 20:12:00.331613 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:00.331245    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" podUID="9f35ea3d-b5b9-47b2-800d-78cc466867af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 20:12:01.197759 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:01.197725    5224 generic.go:356] "Generic (PLEG): container finished" podID="baf37ca7-a400-495d-bce4-cf6613da5355" containerID="db602aede4931e47a7f47abb2c97bb5bf89b56232a5521a9c5290fd681d328bb" exitCode=0
Nov 02 20:12:01.198026 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:01.197802    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" event={"ID":"baf37ca7-a400-495d-bce4-cf6613da5355","Type":"ContainerDied","Data":"db602aede4931e47a7f47abb2c97bb5bf89b56232a5521a9c5290fd681d328bb"}
Nov 02 20:12:01.198309 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:01.198289    5224 scope.go:122] "RemoveContainer" containerID="db602aede4931e47a7f47abb2c97bb5bf89b56232a5521a9c5290fd681d328bb"
Nov 02 20:12:01.199844 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:01.199815    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:12:01.199905 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:01.199885    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:12:01.199941 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:01.199925    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:12:01.330636 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:01.330562    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 20:12:01.330636 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:01.330623    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" podUID="9f35ea3d-b5b9-47b2-800d-78cc466867af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 20:12:02.205782 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:02.205736    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" event={"ID":"baf37ca7-a400-495d-bce4-cf6613da5355","Type":"ContainerStarted","Data":"28da88f973b102a70d58af30b5581b14df16bd78b11b377d4733a9979bf82600"}
Nov 02 20:12:02.335710 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:02.335673    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:12:02.336061 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:02.336043    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:12:02.336810 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:02.336788    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:12:02.578311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:02.578276    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:12:02.579326 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:02.579290    5224 patch_prober.go:29] interesting pod/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 20:12:02.579500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:02.579369    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 20:12:03.578491 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:03.578451    5224 patch_prober.go:29] interesting pod/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 20:12:03.578931 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:03.578511    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 20:12:04.011477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:04.011372    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:04.578494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:04.578448    5224 patch_prober.go:29] interesting pod/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 20:12:04.578923 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:04.578517    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 20:12:05.578230 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:05.578194    5224 patch_prober.go:29] interesting pod/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 20:12:05.578412 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:05.578259    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 20:12:06.578934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:06.578899    5224 patch_prober.go:29] interesting pod/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 20:12:06.579357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:06.578965    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 20:12:07.493954 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:07.493911    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:12:07.579010 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:07.578966    5224 patch_prober.go:29] interesting pod/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 20:12:07.579405 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:07.579040    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 20:12:08.582520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:08.582486    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:12:08.583587 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:08.583566    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:12:09.259281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:09.259244    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:14.563441 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:14.563402    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:15.270927 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:15.270841    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:16.102761 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.102730    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:16.148610 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.148574    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"]
Nov 02 20:12:16.151867 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.151844    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.170669 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.170645    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-home\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.170825 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.170786    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-dshm\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.170911 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.170894    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-model-cache\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.170988 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.170924    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2af629-50fa-4966-87d5-d4d399d844b1-tls-certs\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.171042 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.171003    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-tmp-dir\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.171119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.171100    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q55m\" (UniqueName: \"kubernetes.io/projected/bc2af629-50fa-4966-87d5-d4d399d844b1-kube-api-access-6q55m\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.204704 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.204674    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"]
Nov 02 20:12:16.251505 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.251477    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6a4ab0\"/\"distinct-2-w3-6a4ab0-kserve-self-signed-certs\""
Nov 02 20:12:16.272362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.272326    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q55m\" (UniqueName: \"kubernetes.io/projected/bc2af629-50fa-4966-87d5-d4d399d844b1-kube-api-access-6q55m\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.272551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.272403    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-home\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.272551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.272527    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-dshm\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.272675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.272591    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-model-cache\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.272675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.272612    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2af629-50fa-4966-87d5-d4d399d844b1-tls-certs\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.272675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.272643    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-tmp-dir\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.272919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.272892    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-home\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.272986 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.272934    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-model-cache\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.273122 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.273103    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-tmp-dir\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.274852 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.274829    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-dshm\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.275046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.275028    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2af629-50fa-4966-87d5-d4d399d844b1-tls-certs\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.286732 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.286704    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:16.332711 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.332660    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q55m\" (UniqueName: \"kubernetes.io/projected/bc2af629-50fa-4966-87d5-d4d399d844b1-kube-api-access-6q55m\") pod \"distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") " pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.463858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.463760    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:16.724847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.720646    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:16.913220 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:12:16.912905    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc2af629_50fa_4966_87d5_d4d399d844b1.slice/crio-885b114f1c7194993fd887ac81d6f57331dad5df5b15759a47543350501a41f4 WatchSource:0}: Error finding container 885b114f1c7194993fd887ac81d6f57331dad5df5b15759a47543350501a41f4: Status 404 returned error can't find the container with id 885b114f1c7194993fd887ac81d6f57331dad5df5b15759a47543350501a41f4
Nov 02 20:12:16.923029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:16.921017    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"]
Nov 02 20:12:17.211521 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:17.211456    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:17.309758 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:17.309716    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd" event={"ID":"bc2af629-50fa-4966-87d5-d4d399d844b1","Type":"ContainerStarted","Data":"f6a8b16bcaad2ab0937bd1855b1ce273df18b2a82debacdd4c92a58ddca52e66"}
Nov 02 20:12:17.309951 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:17.309786    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd" event={"ID":"bc2af629-50fa-4966-87d5-d4d399d844b1","Type":"ContainerStarted","Data":"885b114f1c7194993fd887ac81d6f57331dad5df5b15759a47543350501a41f4"}
Nov 02 20:12:17.426950 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:17.426902    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd" podStartSLOduration=1.426888162 podStartE2EDuration="1.426888162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:17.424645417 +0000 UTC m=+1418.422334028" watchObservedRunningTime="2026-11-02 20:12:17.426888162 +0000 UTC m=+1418.424576770"
Nov 02 20:12:17.464457 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:17.464419    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:19.483932 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:19.483857    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:21.503519 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:21.503480    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:21.689932 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:21.689882    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:24.648626 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:24.648583    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:24.821848 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:24.821807    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:25.049507 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:25.049457    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:26.464670 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:26.464639    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:26.477301 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:26.477272    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:27.359245 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:12:27.359208    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:12:27.386000 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:27.385971    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 20:12:28.854604 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:28.854570    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:12:29.532206 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:29.532162    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:30.251397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:30.251364    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:30.480662 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:30.480619    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:31.315636 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:31.315582    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:31.551919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:31.551882    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:35.606701 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:35.606657    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:35.785322 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:35.785282    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:35.982678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:35.982597    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:38.881044 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:38.881012    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:39.065155 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:39.065114    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:39.437602 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:39.437567    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"]
Nov 02 20:12:39.462683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:39.462650    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"
Nov 02 20:12:39.480137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:39.480105    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"]
Nov 02 20:12:39.521781 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:39.521745    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs\" (UID: \"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"
Nov 02 20:12:39.522013 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:39.522000    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6n82\" (UniqueName: \"kubernetes.io/projected/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-kube-api-access-h6n82\") pod \"payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs\" (UID: \"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"
Nov 02 20:12:39.568092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:39.568061    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a3021d\""
Nov 02 20:12:39.568092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:39.568063    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a3021d-dockercfg-gbzn8\""
Nov 02 20:12:39.622872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:39.622843    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6n82\" (UniqueName: \"kubernetes.io/projected/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-kube-api-access-h6n82\") pod \"payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs\" (UID: \"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"
Nov 02 20:12:39.623046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:39.622895    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs\" (UID: \"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"
Nov 02 20:12:39.623649 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:39.623629    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs\" (UID: \"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"
Nov 02 20:12:39.680636 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:39.680601    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6n82\" (UniqueName: \"kubernetes.io/projected/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-kube-api-access-h6n82\") pod \"payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs\" (UID: \"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"
Nov 02 20:12:39.787335 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:39.787303    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"
Nov 02 20:12:40.162571 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:12:40.162537    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9418ffd_ddec_4ef6_ae20_25f9b8875fc9.slice/crio-7db101222987a918134aa06e7805417b1a0b57034763c839f6014d6e887c638e WatchSource:0}: Error finding container 7db101222987a918134aa06e7805417b1a0b57034763c839f6014d6e887c638e: Status 404 returned error can't find the container with id 7db101222987a918134aa06e7805417b1a0b57034763c839f6014d6e887c638e
Nov 02 20:12:40.167045 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:40.167023    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"]
Nov 02 20:12:40.491870 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:40.491755    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs" event={"ID":"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9","Type":"ContainerStarted","Data":"9f3f823fdd3b4c0f4fb7caf695de6f1f9264e55226b737cc530d0facf040683e"}
Nov 02 20:12:40.491870 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:40.491842    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs" event={"ID":"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9","Type":"ContainerStarted","Data":"7db101222987a918134aa06e7805417b1a0b57034763c839f6014d6e887c638e"}
Nov 02 20:12:40.492057 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:40.491988    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"
Nov 02 20:12:40.609590 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:40.609542    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs" podStartSLOduration=1.609527512 podStartE2EDuration="1.609527512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:40.607027103 +0000 UTC m=+1441.604715713" watchObservedRunningTime="2026-11-02 20:12:40.609527512 +0000 UTC m=+1441.607216124"
Nov 02 20:12:41.066062 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:41.066013    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:41.297464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:41.297428    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:41.472466 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:41.472382    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:46.756407 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:46.756366    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:51.502657 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:51.502585    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"
Nov 02 20:12:51.705094 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:51.705054    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:52.734267 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:52.734224    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:53.608201 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:53.608169    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:54.143907 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.143876    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"]
Nov 02 20:12:54.147167 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.147150    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.255708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.255675    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"]
Nov 02 20:12:54.315423 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.315386    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.315566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.315509    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.315566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.315546    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54740835-67a3-418f-a090-18345e77f536-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.315654 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.315621    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.315692 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.315666    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.316039 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.316017    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.316164 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.316040    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54740835-67a3-418f-a090-18345e77f536-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.316225 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.316167    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgff\" (UniqueName: \"kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-kube-api-access-bfgff\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.316273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.316232    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.417563 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.417467    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.417716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.417566    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.417716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.417592    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54740835-67a3-418f-a090-18345e77f536-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.417716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.417618    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.417911 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.417761    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.417970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.417953    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.418024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.417985    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.418024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.418009    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54740835-67a3-418f-a090-18345e77f536-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.418024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.418012    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.418237 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.418083    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgff\" (UniqueName: \"kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-kube-api-access-bfgff\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.418237 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.418123    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.418396 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.418373    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.418505 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.418489    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.418818 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.418796    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54740835-67a3-418f-a090-18345e77f536-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.420051 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.420031    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.420218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.420201    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54740835-67a3-418f-a090-18345e77f536-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.479310 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.479277    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.479530 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.479507    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgff\" (UniqueName: \"kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-kube-api-access-bfgff\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:54.759788 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:54.759691    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:55.222902 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:12:55.222871    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54740835_67a3_418f_a090_18345e77f536.slice/crio-c759155346778165ba610dddf1ca543349b2dd5fc4aede16c14646de3d7c438d WatchSource:0}: Error finding container c759155346778165ba610dddf1ca543349b2dd5fc4aede16c14646de3d7c438d: Status 404 returned error can't find the container with id c759155346778165ba610dddf1ca543349b2dd5fc4aede16c14646de3d7c438d
Nov 02 20:12:55.225555 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:55.225520    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:12:55.225649 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:55.225584    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:12:55.225649 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:55.225613    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:12:55.227555 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:55.227531    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"]
Nov 02 20:12:55.595490 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:55.595450    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95" event={"ID":"54740835-67a3-418f-a090-18345e77f536","Type":"ContainerStarted","Data":"9e42caf44b03d8057bbd63fe4a3b997bd05b2ac17a4a8854a5695a10f9d93f69"}
Nov 02 20:12:55.595699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:55.595510    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95" event={"ID":"54740835-67a3-418f-a090-18345e77f536","Type":"ContainerStarted","Data":"c759155346778165ba610dddf1ca543349b2dd5fc4aede16c14646de3d7c438d"}
Nov 02 20:12:55.717710 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:55.717656    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95" podStartSLOduration=1.717643831 podStartE2EDuration="1.717643831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:55.717455509 +0000 UTC m=+1456.715144117" watchObservedRunningTime="2026-11-02 20:12:55.717643831 +0000 UTC m=+1456.715332440"
Nov 02 20:12:55.760779 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:55.760731    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:12:56.774819 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:56.774067    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 20:12:56.774819 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:56.774152    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95" podUID="54740835-67a3-418f-a090-18345e77f536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 20:12:56.883863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:56.883822    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:12:57.389932 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:12:57.389890    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:12:57.763052 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:57.763018    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 20:12:57.763330 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:57.763297    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95" podUID="54740835-67a3-418f-a090-18345e77f536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 20:12:58.767792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:58.763216    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 20:12:58.767792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:58.763281    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95" podUID="54740835-67a3-418f-a090-18345e77f536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 20:12:59.774260 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:59.774223    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 20:12:59.774924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:12:59.774881    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95" podUID="54740835-67a3-418f-a090-18345e77f536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 20:13:00.767675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:00.767639    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 20:13:00.767998 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:00.767966    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95" podUID="54740835-67a3-418f-a090-18345e77f536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 20:13:01.760287 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:01.760247    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 20:13:01.760834 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:01.760299    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95" podUID="54740835-67a3-418f-a090-18345e77f536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 20:13:01.903699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:01.903663    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:03.681986 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:03.681947    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:13:03.682488 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:03.682228    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:13:03.683376 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:03.683353    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:13:04.487323 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.487290    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"]
Nov 02 20:13:04.490374 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.490357    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"
Nov 02 20:13:04.543211 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.543180    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"]
Nov 02 20:13:04.544341 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.544318    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:13:04.544476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.544342    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:13:04.544584 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.544567    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:13:04.544664 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.544653    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:13:04.620171 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.620138    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09d1ceaf-glh4c\" (UID: \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"
Nov 02 20:13:04.620353 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.620336    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc848\" (UniqueName: \"kubernetes.io/projected/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-kube-api-access-jc848\") pod \"maas-api-revoke-sub-09d1ceaf-glh4c\" (UID: \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"
Nov 02 20:13:04.620498 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.620482    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09d1ceaf-glh4c\" (UID: \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"
Nov 02 20:13:04.722136 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.722105    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09d1ceaf-glh4c\" (UID: \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"
Nov 02 20:13:04.722609 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.722239    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc848\" (UniqueName: \"kubernetes.io/projected/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-kube-api-access-jc848\") pod \"maas-api-revoke-sub-09d1ceaf-glh4c\" (UID: \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"
Nov 02 20:13:04.722609 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.722343    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09d1ceaf-glh4c\" (UID: \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"
Nov 02 20:13:04.722939 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.722913    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09d1ceaf-glh4c\" (UID: \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"
Nov 02 20:13:04.724579 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.724561    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09d1ceaf-glh4c\" (UID: \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"
Nov 02 20:13:04.780374 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.780348    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc848\" (UniqueName: \"kubernetes.io/projected/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-kube-api-access-jc848\") pod \"maas-api-revoke-sub-09d1ceaf-glh4c\" (UID: \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"
Nov 02 20:13:04.803756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:04.803728    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"
Nov 02 20:13:05.173840 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:13:05.173811    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d08a244_6b6f_4fb7_8b9a_8894a6bc1f4f.slice/crio-9a387cf7e753e739b2534cdd0cd0ca291c3460a83b52ed625bccdaf5df4ec9dd WatchSource:0}: Error finding container 9a387cf7e753e739b2534cdd0cd0ca291c3460a83b52ed625bccdaf5df4ec9dd: Status 404 returned error can't find the container with id 9a387cf7e753e739b2534cdd0cd0ca291c3460a83b52ed625bccdaf5df4ec9dd
Nov 02 20:13:05.179270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:05.179237    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"]
Nov 02 20:13:05.669998 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:05.669967    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c" event={"ID":"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f","Type":"ContainerStarted","Data":"7311f2d676773f8c3f7d57d1a31fe2c419f3a25c24df218b0c40b2860b115fce"}
Nov 02 20:13:05.670183 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:05.670019    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c" event={"ID":"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f","Type":"ContainerStarted","Data":"9a387cf7e753e739b2534cdd0cd0ca291c3460a83b52ed625bccdaf5df4ec9dd"}
Nov 02 20:13:05.787024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:05.786969    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c" podStartSLOduration=1.786955586 podStartE2EDuration="1.786955586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:05.784615152 +0000 UTC m=+1466.782303761" watchObservedRunningTime="2026-11-02 20:13:05.786955586 +0000 UTC m=+1466.784644195"
Nov 02 20:13:06.678618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:06.678586    5224 generic.go:356] "Generic (PLEG): container finished" podID="4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f" containerID="7311f2d676773f8c3f7d57d1a31fe2c419f3a25c24df218b0c40b2860b115fce" exitCode=0
Nov 02 20:13:06.678831 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:06.678630    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c" event={"ID":"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f","Type":"ContainerDied","Data":"7311f2d676773f8c3f7d57d1a31fe2c419f3a25c24df218b0c40b2860b115fce"}
Nov 02 20:13:06.864523 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:06.864487    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:07.830079 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:07.830053    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"
Nov 02 20:13:07.963752 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:07.963665    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-openshift-ca\") pod \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\" (UID: \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\") "
Nov 02 20:13:07.964218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:07.963814    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-kube-api-access-jc848\" (UniqueName: \"kubernetes.io/projected/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-kube-api-access-jc848\") pod \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\" (UID: \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\") "
Nov 02 20:13:07.964218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:07.963971    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-xks-ca\") pod \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\" (UID: \"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f\") "
Nov 02 20:13:07.964218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:07.964052    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-openshift-ca" pod "4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f" (UID: "4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:13:07.964575 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:07.964548    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:07.966078 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:07.966057    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-xks-ca" pod "4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f" (UID: "4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:13:07.966322 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:07.966305    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-kube-api-access-jc848" pod "4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f" (UID: "4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f"). InnerVolumeSpecName "kube-api-access-jc848". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:13:08.065808 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:08.065757    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc848\" (UniqueName: \"kubernetes.io/projected/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-kube-api-access-jc848\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:08.065808 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:08.065808    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:08.693835 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:08.693795    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c" event={"ID":"4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f","Type":"ContainerDied","Data":"9a387cf7e753e739b2534cdd0cd0ca291c3460a83b52ed625bccdaf5df4ec9dd"}
Nov 02 20:13:08.694022 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:08.693860    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a387cf7e753e739b2534cdd0cd0ca291c3460a83b52ed625bccdaf5df4ec9dd"
Nov 02 20:13:08.694022 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:08.693807    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"
Nov 02 20:13:11.916173 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:11.916126    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:16.706044 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:16.706008    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:16.922926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:16.922882    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:17.193696 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:17.193660    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:21.765333 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:21.765297    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:25.257319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:25.257277    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:13:26.747339 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:26.747297    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:27.359214 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:13:27.359181    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:13:31.491803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:31.491739    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:31.760231 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:31.760144    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:31.933449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:31.933414    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:35.592917 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:35.592886    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7522-429444"]
Nov 02 20:13:35.593469 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:35.593455    5224 cpu_manager.go:414] "containerMap: removing container" podUID="4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f" containerName="revoke-keys"
Nov 02 20:13:35.593578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:35.593470    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f" containerName="revoke-keys"
Nov 02 20:13:35.593616 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:35.593585    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f" containerName="revoke-keys"
Nov 02 20:13:35.597075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:35.597058    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-429444"
Nov 02 20:13:35.648317 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:35.648281    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7522-429444"]
Nov 02 20:13:35.695194 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:35.695167    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bkzxn\""
Nov 02 20:13:35.715621 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:35.715591    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq58r\" (UniqueName: \"kubernetes.io/projected/e456b47e-652f-4b14-9f24-0bf9fd9547ce-kube-api-access-qq58r\") pod \"test-curl-7522-429444\" (UID: \"e456b47e-652f-4b14-9f24-0bf9fd9547ce\") " pod="opendatahub/test-curl-7522-429444"
Nov 02 20:13:35.817235 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:35.817205    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq58r\" (UniqueName: \"kubernetes.io/projected/e456b47e-652f-4b14-9f24-0bf9fd9547ce-kube-api-access-qq58r\") pod \"test-curl-7522-429444\" (UID: \"e456b47e-652f-4b14-9f24-0bf9fd9547ce\") " pod="opendatahub/test-curl-7522-429444"
Nov 02 20:13:35.874965 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:35.874894    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq58r\" (UniqueName: \"kubernetes.io/projected/e456b47e-652f-4b14-9f24-0bf9fd9547ce-kube-api-access-qq58r\") pod \"test-curl-7522-429444\" (UID: \"e456b47e-652f-4b14-9f24-0bf9fd9547ce\") " pod="opendatahub/test-curl-7522-429444"
Nov 02 20:13:35.909301 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:35.909264    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-429444"
Nov 02 20:13:36.270745 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:13:36.270718    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode456b47e_652f_4b14_9f24_0bf9fd9547ce.slice/crio-2aa7734c6e6e8eab1763b0a37751460e24fce511034a5fc004bded1613c3eff6 WatchSource:0}: Error finding container 2aa7734c6e6e8eab1763b0a37751460e24fce511034a5fc004bded1613c3eff6: Status 404 returned error can't find the container with id 2aa7734c6e6e8eab1763b0a37751460e24fce511034a5fc004bded1613c3eff6
Nov 02 20:13:36.272554 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:36.272538    5224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:13:36.277522 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:36.277496    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7522-429444"]
Nov 02 20:13:36.644356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:36.644265    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:36.891793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:36.891735    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7522-429444" event={"ID":"e456b47e-652f-4b14-9f24-0bf9fd9547ce","Type":"ContainerStarted","Data":"2aa7734c6e6e8eab1763b0a37751460e24fce511034a5fc004bded1613c3eff6"}
Nov 02 20:13:37.050120 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:37.050089    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"]
Nov 02 20:13:37.101413 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:37.101384    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c727c0-zgh4z"]
Nov 02 20:13:37.168238 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:37.168207    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="644fb6e2-8934-4031-a125-8c9fceea7397" path="/var/lib/kubelet/pods/644fb6e2-8934-4031-a125-8c9fceea7397/volumes"
Nov 02 20:13:39.916491 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:39.916446    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7522-429444" event={"ID":"e456b47e-652f-4b14-9f24-0bf9fd9547ce","Type":"ContainerStarted","Data":"cdd76bb5aef7dc5990600ea098fbce4c8488f1b201ca6ebad6a4469a1f13af80"}
Nov 02 20:13:40.034204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:40.034153    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7522-429444" podStartSLOduration=2.145972695 podStartE2EDuration="5.0341399s" totalImagesPullingTime="2.888167205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:40.030916374 +0000 UTC m=+1501.028604983" watchObservedRunningTime="2026-11-02 20:13:40.0341399 +0000 UTC m=+1501.031828509"
Nov 02 20:13:41.537681 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:41.537650    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7522-429444"]
Nov 02 20:13:41.544511 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:41.544475    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7522-429444"]
Nov 02 20:13:41.553576 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:41.553543    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:41.929783 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:41.929683    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7522-429444" podUID="e456b47e-652f-4b14-9f24-0bf9fd9547ce" containerName="test-curl-7522-429444" containerID="cri-o://cdd76bb5aef7dc5990600ea098fbce4c8488f1b201ca6ebad6a4469a1f13af80" gracePeriod=2
Nov 02 20:13:41.982296 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:13:41.982253    5224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7522-429444\" is forbidden: User \"system:node:ip-10-0-134-196.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-196.us-west-2.compute.internal' and this object" podUID="e456b47e-652f-4b14-9f24-0bf9fd9547ce" pod="opendatahub/test-curl-7522-429444"
Nov 02 20:13:42.497226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.497190    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"]
Nov 02 20:13:42.497823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.497808    5224 cpu_manager.go:414] "containerMap: removing container" podUID="e456b47e-652f-4b14-9f24-0bf9fd9547ce" containerName="test-curl-7522-429444"
Nov 02 20:13:42.497892 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.497825    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e456b47e-652f-4b14-9f24-0bf9fd9547ce" containerName="test-curl-7522-429444"
Nov 02 20:13:42.497937 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.497926    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="e456b47e-652f-4b14-9f24-0bf9fd9547ce" containerName="test-curl-7522-429444"
Nov 02 20:13:42.500984 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.500966    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"
Nov 02 20:13:42.554014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.553980    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"]
Nov 02 20:13:42.601434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.601399    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:13:42.601434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.601421    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:13:42.601434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.601406    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:13:42.601655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.601438    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:13:42.606036 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.606011    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j72k\" (UniqueName: \"kubernetes.io/projected/fbd3729b-4162-427e-b40f-3d74e21b820e-kube-api-access-7j72k\") pod \"maas-api-revoke-keys-e2e-mt-5e04421b-rzchc\" (UID: \"fbd3729b-4162-427e-b40f-3d74e21b820e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"
Nov 02 20:13:42.606239 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.606220    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5e04421b-rzchc\" (UID: \"fbd3729b-4162-427e-b40f-3d74e21b820e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"
Nov 02 20:13:42.606359 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.606342    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5e04421b-rzchc\" (UID: \"fbd3729b-4162-427e-b40f-3d74e21b820e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"
Nov 02 20:13:42.707875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.707845    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5e04421b-rzchc\" (UID: \"fbd3729b-4162-427e-b40f-3d74e21b820e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"
Nov 02 20:13:42.708040 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.707909    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j72k\" (UniqueName: \"kubernetes.io/projected/fbd3729b-4162-427e-b40f-3d74e21b820e-kube-api-access-7j72k\") pod \"maas-api-revoke-keys-e2e-mt-5e04421b-rzchc\" (UID: \"fbd3729b-4162-427e-b40f-3d74e21b820e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"
Nov 02 20:13:42.708104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.708053    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5e04421b-rzchc\" (UID: \"fbd3729b-4162-427e-b40f-3d74e21b820e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"
Nov 02 20:13:42.708479 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.708458    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5e04421b-rzchc\" (UID: \"fbd3729b-4162-427e-b40f-3d74e21b820e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"
Nov 02 20:13:42.710424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.710398    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5e04421b-rzchc\" (UID: \"fbd3729b-4162-427e-b40f-3d74e21b820e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"
Nov 02 20:13:42.714045 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:13:42.714017    5224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7522-429444\" is forbidden: User \"system:node:ip-10-0-134-196.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-196.us-west-2.compute.internal' and this object" podUID="e456b47e-652f-4b14-9f24-0bf9fd9547ce" pod="opendatahub/test-curl-7522-429444"
Nov 02 20:13:42.765354 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.765328    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j72k\" (UniqueName: \"kubernetes.io/projected/fbd3729b-4162-427e-b40f-3d74e21b820e-kube-api-access-7j72k\") pod \"maas-api-revoke-keys-e2e-mt-5e04421b-rzchc\" (UID: \"fbd3729b-4162-427e-b40f-3d74e21b820e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"
Nov 02 20:13:42.813723 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.813697    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"
Nov 02 20:13:42.899104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:42.899071    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:43.185163 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:13:43.185130    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd3729b_4162_427e_b40f_3d74e21b820e.slice/crio-694a1adca14e8a5258666219ef1653b48d0629c8629e62293af97e3cd6d1c309 WatchSource:0}: Error finding container 694a1adca14e8a5258666219ef1653b48d0629c8629e62293af97e3cd6d1c309: Status 404 returned error can't find the container with id 694a1adca14e8a5258666219ef1653b48d0629c8629e62293af97e3cd6d1c309
Nov 02 20:13:43.189551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.189526    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"]
Nov 02 20:13:43.355975 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.355945    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"]
Nov 02 20:13:43.359859 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.359840    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.478609 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.478529    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-15203f-openshift-default-dockercfg-4q7b2\""
Nov 02 20:13:43.480860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.480830    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-podinfo\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.481158 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.481108    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-certs\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.481308 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.481291    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-socket\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.481423 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.481407    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6vsr\" (UniqueName: \"kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-kube-api-access-h6vsr\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.481555 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.481541    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-envoy\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.481723 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.481707    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-credential-socket\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.481907 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.481874    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-data\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.481989 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.481940    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df701f2f-911b-40e0-9e88-a66aa003ee45-istiod-ca-cert\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.482050 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.481998    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-token\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.484284 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.484256    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"]
Nov 02 20:13:43.500835 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.493067    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:43.583358 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.583328    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-certs\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.583811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.583371    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-socket\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.583811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.583396    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6vsr\" (UniqueName: \"kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-kube-api-access-h6vsr\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.583811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.583452    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-envoy\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.583811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.583536    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-credential-socket\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.583811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.583649    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-data\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.583811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.583696    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df701f2f-911b-40e0-9e88-a66aa003ee45-istiod-ca-cert\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.583811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.583785    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-token\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.583811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.583799    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-socket\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.584246 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.583958    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-podinfo\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.584246 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.584105    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-certs\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.584421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.584398    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-data\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.584533 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.584506    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df701f2f-911b-40e0-9e88-a66aa003ee45-istiod-ca-cert\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.584708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.584681    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-credential-socket\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.586744 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.586723    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-envoy\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.587052 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.587034    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-podinfo\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.642362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.642329    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-token\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.642658 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.642626    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6vsr\" (UniqueName: \"kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-kube-api-access-h6vsr\") pod \"e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") " pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.673450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.673414    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:43.962380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.962342    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc" event={"ID":"fbd3729b-4162-427e-b40f-3d74e21b820e","Type":"ContainerStarted","Data":"445185083beb114ddc71bae93203b6d26195587782c4da87466863e0a66a246f"}
Nov 02 20:13:43.962497 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.962407    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc" event={"ID":"fbd3729b-4162-427e-b40f-3d74e21b820e","Type":"ContainerStarted","Data":"694a1adca14e8a5258666219ef1653b48d0629c8629e62293af97e3cd6d1c309"}
Nov 02 20:13:43.965276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:43.965220    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:44.079313 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.079255    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc" podStartSLOduration=2.079234454 podStartE2EDuration="2.079234454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:44.076349075 +0000 UTC m=+1505.074037686" watchObservedRunningTime="2026-11-02 20:13:44.079234454 +0000 UTC m=+1505.076923063"
Nov 02 20:13:44.120422 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.120379    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"]
Nov 02 20:13:44.124236 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:13:44.124212    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf701f2f_911b_40e0_9e88_a66aa003ee45.slice/crio-f972f31543d3a42adac5d4728d4231b5be247ab2ab00d93668ba6e41944658ac WatchSource:0}: Error finding container f972f31543d3a42adac5d4728d4231b5be247ab2ab00d93668ba6e41944658ac: Status 404 returned error can't find the container with id f972f31543d3a42adac5d4728d4231b5be247ab2ab00d93668ba6e41944658ac
Nov 02 20:13:44.127424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.127389    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:13:44.127531 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.127472    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:13:44.127531 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.127519    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:13:44.268261 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.268234    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-429444"
Nov 02 20:13:44.409266 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.409180    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e456b47e-652f-4b14-9f24-0bf9fd9547ce-kube-api-access-qq58r\" (UniqueName: \"kubernetes.io/projected/e456b47e-652f-4b14-9f24-0bf9fd9547ce-kube-api-access-qq58r\") pod \"e456b47e-652f-4b14-9f24-0bf9fd9547ce\" (UID: \"e456b47e-652f-4b14-9f24-0bf9fd9547ce\") "
Nov 02 20:13:44.411429 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.411402    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e456b47e-652f-4b14-9f24-0bf9fd9547ce-kube-api-access-qq58r" pod "e456b47e-652f-4b14-9f24-0bf9fd9547ce" (UID: "e456b47e-652f-4b14-9f24-0bf9fd9547ce"). InnerVolumeSpecName "kube-api-access-qq58r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:13:44.510635 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.510604    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq58r\" (UniqueName: \"kubernetes.io/projected/e456b47e-652f-4b14-9f24-0bf9fd9547ce-kube-api-access-qq58r\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:44.976049 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.975953    5224 generic.go:356] "Generic (PLEG): container finished" podID="e456b47e-652f-4b14-9f24-0bf9fd9547ce" containerID="cdd76bb5aef7dc5990600ea098fbce4c8488f1b201ca6ebad6a4469a1f13af80" exitCode=137
Nov 02 20:13:44.976484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.976211    5224 scope.go:122] "RemoveContainer" containerID="cdd76bb5aef7dc5990600ea098fbce4c8488f1b201ca6ebad6a4469a1f13af80"
Nov 02 20:13:44.976484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.976417    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-429444"
Nov 02 20:13:44.979688 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.979477    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6" event={"ID":"df701f2f-911b-40e0-9e88-a66aa003ee45","Type":"ContainerStarted","Data":"1c1f67f6f0d9510a4ada06d708f1c9b49b17fe13282f8bb1355d9419a8a0e696"}
Nov 02 20:13:44.979688 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.979592    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6" event={"ID":"df701f2f-911b-40e0-9e88-a66aa003ee45","Type":"ContainerStarted","Data":"f972f31543d3a42adac5d4728d4231b5be247ab2ab00d93668ba6e41944658ac"}
Nov 02 20:13:44.982167 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.982142    5224 generic.go:356] "Generic (PLEG): container finished" podID="fbd3729b-4162-427e-b40f-3d74e21b820e" containerID="445185083beb114ddc71bae93203b6d26195587782c4da87466863e0a66a246f" exitCode=0
Nov 02 20:13:44.982304 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.982175    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc" event={"ID":"fbd3729b-4162-427e-b40f-3d74e21b820e","Type":"ContainerDied","Data":"445185083beb114ddc71bae93203b6d26195587782c4da87466863e0a66a246f"}
Nov 02 20:13:44.989492 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.989468    5224 scope.go:122] "RemoveContainer" containerID="cdd76bb5aef7dc5990600ea098fbce4c8488f1b201ca6ebad6a4469a1f13af80"
Nov 02 20:13:44.989811 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:13:44.989787    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd76bb5aef7dc5990600ea098fbce4c8488f1b201ca6ebad6a4469a1f13af80\": container with ID starting with cdd76bb5aef7dc5990600ea098fbce4c8488f1b201ca6ebad6a4469a1f13af80 not found: ID does not exist" containerID="cdd76bb5aef7dc5990600ea098fbce4c8488f1b201ca6ebad6a4469a1f13af80"
Nov 02 20:13:44.989875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:44.989818    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd76bb5aef7dc5990600ea098fbce4c8488f1b201ca6ebad6a4469a1f13af80"} err="failed to get container status \"cdd76bb5aef7dc5990600ea098fbce4c8488f1b201ca6ebad6a4469a1f13af80\": rpc error: code = NotFound desc = could not find container \"cdd76bb5aef7dc5990600ea098fbce4c8488f1b201ca6ebad6a4469a1f13af80\": container with ID starting with cdd76bb5aef7dc5990600ea098fbce4c8488f1b201ca6ebad6a4469a1f13af80 not found: ID does not exist"
Nov 02 20:13:45.099348 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:45.099299    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6" podStartSLOduration=2.09928686 podStartE2EDuration="2.09928686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:45.096931611 +0000 UTC m=+1506.094620219" watchObservedRunningTime="2026-11-02 20:13:45.09928686 +0000 UTC m=+1506.096975469"
Nov 02 20:13:45.149171 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:13:45.149136    5224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7522-429444\" is forbidden: User \"system:node:ip-10-0-134-196.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-196.us-west-2.compute.internal' and this object" podUID="e456b47e-652f-4b14-9f24-0bf9fd9547ce" pod="opendatahub/test-curl-7522-429444"
Nov 02 20:13:45.167956 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:45.167927    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e456b47e-652f-4b14-9f24-0bf9fd9547ce" path="/var/lib/kubelet/pods/e456b47e-652f-4b14-9f24-0bf9fd9547ce/volumes"
Nov 02 20:13:45.674215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:45.674185    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:45.679561 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:45.679540    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:45.990874 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:45.990413    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:45.991272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:45.991253    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:13:46.276759 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.276731    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"
Nov 02 20:13:46.358801 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.358749    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-xks-ca\") pod \"fbd3729b-4162-427e-b40f-3d74e21b820e\" (UID: \"fbd3729b-4162-427e-b40f-3d74e21b820e\") "
Nov 02 20:13:46.358960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.358840    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-openshift-ca\") pod \"fbd3729b-4162-427e-b40f-3d74e21b820e\" (UID: \"fbd3729b-4162-427e-b40f-3d74e21b820e\") "
Nov 02 20:13:46.358960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.358932    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd3729b-4162-427e-b40f-3d74e21b820e-kube-api-access-7j72k\" (UniqueName: \"kubernetes.io/projected/fbd3729b-4162-427e-b40f-3d74e21b820e-kube-api-access-7j72k\") pod \"fbd3729b-4162-427e-b40f-3d74e21b820e\" (UID: \"fbd3729b-4162-427e-b40f-3d74e21b820e\") "
Nov 02 20:13:46.359246 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.359221    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-openshift-ca" pod "fbd3729b-4162-427e-b40f-3d74e21b820e" (UID: "fbd3729b-4162-427e-b40f-3d74e21b820e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:13:46.359467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.359450    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:46.360758 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.360734    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd3729b-4162-427e-b40f-3d74e21b820e-kube-api-access-7j72k" pod "fbd3729b-4162-427e-b40f-3d74e21b820e" (UID: "fbd3729b-4162-427e-b40f-3d74e21b820e"). InnerVolumeSpecName "kube-api-access-7j72k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:13:46.360882 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.360809    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-xks-ca" pod "fbd3729b-4162-427e-b40f-3d74e21b820e" (UID: "fbd3729b-4162-427e-b40f-3d74e21b820e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:13:46.391692 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.391662    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:46.460192 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.460163    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd3729b-4162-427e-b40f-3d74e21b820e-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:46.460192 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.460189    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j72k\" (UniqueName: \"kubernetes.io/projected/fbd3729b-4162-427e-b40f-3d74e21b820e-kube-api-access-7j72k\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:46.999364 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.999335    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"
Nov 02 20:13:46.999838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.999398    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc" event={"ID":"fbd3729b-4162-427e-b40f-3d74e21b820e","Type":"ContainerDied","Data":"694a1adca14e8a5258666219ef1653b48d0629c8629e62293af97e3cd6d1c309"}
Nov 02 20:13:46.999838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:46.999464    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="694a1adca14e8a5258666219ef1653b48d0629c8629e62293af97e3cd6d1c309"
Nov 02 20:13:47.184186 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:47.184155    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:48.585547 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:48.585496    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:48.817694 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:48.817661    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:49.003078 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:49.002988    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:50.803655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:50.803615    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:51.464393 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:51.464356    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:51.929625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:51.929592    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"]
Nov 02 20:13:51.931276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:51.931244    5224 cpu_manager.go:414] "containerMap: removing container" podUID="fbd3729b-4162-427e-b40f-3d74e21b820e" containerName="revoke-keys"
Nov 02 20:13:51.931276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:51.931274    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd3729b-4162-427e-b40f-3d74e21b820e" containerName="revoke-keys"
Nov 02 20:13:51.931685 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:51.931669    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd3729b-4162-427e-b40f-3d74e21b820e" containerName="revoke-keys"
Nov 02 20:13:51.941356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:51.941331    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:51.995642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:51.995580    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e0aa5c-openshift-default-dockercfg-v8nrb\""
Nov 02 20:13:52.057868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.042211    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-credential-socket\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.057868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.042272    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-data\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.057868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.042308    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84g8t\" (UniqueName: \"kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-kube-api-access-84g8t\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.057868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.042371    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-podinfo\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.057868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.042404    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c3b61e-5330-4cab-8e14-235fefda4c8c-istiod-ca-cert\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.057868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.042439    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-socket\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.057868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.042487    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-envoy\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.057868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.042692    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-token\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.057868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.042723    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-certs\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.123818 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.120515    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"]
Nov 02 20:13:52.130800 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.125830    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:52.179520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.178632    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-credential-socket\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.179520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.178674    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-data\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.179520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.178696    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84g8t\" (UniqueName: \"kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-kube-api-access-84g8t\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.179520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.178729    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-podinfo\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.179520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.178754    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c3b61e-5330-4cab-8e14-235fefda4c8c-istiod-ca-cert\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.179520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.178802    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-socket\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.179520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.178840    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-envoy\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.179520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.178946    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-token\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.179520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.178963    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-certs\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.179520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.179359    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-certs\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.180515 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.180447    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-credential-socket\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.180708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.180685    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-data\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.185640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.181480    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-socket\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.185640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.182092    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c3b61e-5330-4cab-8e14-235fefda4c8c-istiod-ca-cert\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.185640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.184571    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-envoy\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.188525 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.188497    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-podinfo\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.239237 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.239202    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84g8t\" (UniqueName: \"kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-kube-api-access-84g8t\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.239390 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.239313    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-token\") pod \"e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") " pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.256898 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.256874    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:52.452710 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.452623    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:52.701717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.701683    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"]
Nov 02 20:13:52.707288 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:13:52.707259    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c3b61e_5330_4cab_8e14_235fefda4c8c.slice/crio-8d124966e542b44f32f3f945534b8841906b95189aceed8eed4ef539fb75cc84 WatchSource:0}: Error finding container 8d124966e542b44f32f3f945534b8841906b95189aceed8eed4ef539fb75cc84: Status 404 returned error can't find the container with id 8d124966e542b44f32f3f945534b8841906b95189aceed8eed4ef539fb75cc84
Nov 02 20:13:52.710376 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.710335    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:13:52.710485 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.710402    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:13:52.710485 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.710433    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:13:52.778385 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.778301    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:52.957947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:52.957855    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:53.064793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.064106    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"]
Nov 02 20:13:53.080236 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.080199    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38209362-tqvfw"]
Nov 02 20:13:53.142802 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.141610    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:53.142802 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.142031    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" event={"ID":"40c3b61e-5330-4cab-8e14-235fefda4c8c","Type":"ContainerStarted","Data":"ed5c10f39f9918a7a23cf82917b7196d5c7960eb0a164ad738b4e6d962f450a8"}
Nov 02 20:13:53.142802 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.142091    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" event={"ID":"40c3b61e-5330-4cab-8e14-235fefda4c8c","Type":"ContainerStarted","Data":"8d124966e542b44f32f3f945534b8841906b95189aceed8eed4ef539fb75cc84"}
Nov 02 20:13:53.177601 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.177569    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806ccf88-3a32-4615-823a-b9db72fd51f4" path="/var/lib/kubelet/pods/806ccf88-3a32-4615-823a-b9db72fd51f4/volumes"
Nov 02 20:13:53.259172 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.259084    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:53.259349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.259193    5224 patch_prober.go:29] interesting pod/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 20:13:53.259349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.259225    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 20:13:53.269225 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.268741    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" podStartSLOduration=2.268722668 podStartE2EDuration="2.268722668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:53.263254648 +0000 UTC m=+1514.260943258" watchObservedRunningTime="2026-11-02 20:13:53.268722668 +0000 UTC m=+1514.266411278"
Nov 02 20:13:53.279552 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.278755    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"]
Nov 02 20:13:53.283364 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.283338    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"
Nov 02 20:13:53.336953 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.336917    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"]
Nov 02 20:13:53.338278 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.338192    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-15203f\""
Nov 02 20:13:53.341792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.338821    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-15203f-dockercfg-sq676\""
Nov 02 20:13:53.418994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.414975    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmcf9\" (UniqueName: \"kubernetes.io/projected/77cae86a-4c75-452c-8ac5-7845f9fbb985-kube-api-access-tmcf9\") pod \"payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5\" (UID: \"77cae86a-4c75-452c-8ac5-7845f9fbb985\") " pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"
Nov 02 20:13:53.419636 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.419604    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77cae86a-4c75-452c-8ac5-7845f9fbb985-plugins-config-volume\") pod \"payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5\" (UID: \"77cae86a-4c75-452c-8ac5-7845f9fbb985\") " pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"
Nov 02 20:13:53.521521 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.521482    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77cae86a-4c75-452c-8ac5-7845f9fbb985-plugins-config-volume\") pod \"payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5\" (UID: \"77cae86a-4c75-452c-8ac5-7845f9fbb985\") " pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"
Nov 02 20:13:53.521709 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.521614    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmcf9\" (UniqueName: \"kubernetes.io/projected/77cae86a-4c75-452c-8ac5-7845f9fbb985-kube-api-access-tmcf9\") pod \"payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5\" (UID: \"77cae86a-4c75-452c-8ac5-7845f9fbb985\") " pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"
Nov 02 20:13:53.522378 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.522354    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77cae86a-4c75-452c-8ac5-7845f9fbb985-plugins-config-volume\") pod \"payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5\" (UID: \"77cae86a-4c75-452c-8ac5-7845f9fbb985\") " pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"
Nov 02 20:13:53.579728 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.579690    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmcf9\" (UniqueName: \"kubernetes.io/projected/77cae86a-4c75-452c-8ac5-7845f9fbb985-kube-api-access-tmcf9\") pod \"payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5\" (UID: \"77cae86a-4c75-452c-8ac5-7845f9fbb985\") " pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"
Nov 02 20:13:53.602292 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.602258    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"
Nov 02 20:13:53.980086 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.979996    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:53.989134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:53.989108    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"]
Nov 02 20:13:53.997850 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:13:53.997824    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cae86a_4c75_452c_8ac5_7845f9fbb985.slice/crio-6be8d36258d08c651c0affe57da9e307dea3d1e2e2ce3d428384a5dec47608e6 WatchSource:0}: Error finding container 6be8d36258d08c651c0affe57da9e307dea3d1e2e2ce3d428384a5dec47608e6: Status 404 returned error can't find the container with id 6be8d36258d08c651c0affe57da9e307dea3d1e2e2ce3d428384a5dec47608e6
Nov 02 20:13:54.002826 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:54.002762    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"]
Nov 02 20:13:54.150014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:54.149976    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5" event={"ID":"77cae86a-4c75-452c-8ac5-7845f9fbb985","Type":"ContainerStarted","Data":"2bfd774f43116e88df40fc2fd70415dd13a06ab4b83cbc0511f67bf0efe42789"}
Nov 02 20:13:54.150196 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:54.150039    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5" event={"ID":"77cae86a-4c75-452c-8ac5-7845f9fbb985","Type":"ContainerStarted","Data":"6be8d36258d08c651c0affe57da9e307dea3d1e2e2ce3d428384a5dec47608e6"}
Nov 02 20:13:54.150250 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:54.150194    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"
Nov 02 20:13:54.261694 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:54.261621    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:54.267853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:54.267811    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5" podStartSLOduration=1.267799163 podStartE2EDuration="1.267799163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:54.266270661 +0000 UTC m=+1515.263959269" watchObservedRunningTime="2026-11-02 20:13:54.267799163 +0000 UTC m=+1515.265487768"
Nov 02 20:13:54.390167 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:54.390129    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:54.565086 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:54.565054    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:55.157891 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:55.157860    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:55.158943 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:55.158913    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:13:56.354713 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:56.354658    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:56.543846 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:56.543811    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:56.773567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:56.773535    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:57.358905 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:13:57.358873    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:13:58.674590 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:58.674507    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:13:58.989742 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:13:58.989654    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb" podUID="20d012b3-b7bc-45da-99c6-741d3d2c01cd" containerName="payload-processing" containerID="cri-o://552d93288cc854de6ead763a01924f1b6331865f74910f7f1205f25f54e60f54" gracePeriod=25
Nov 02 20:14:00.147629 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:00.147599    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:00.385484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:00.385445    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:04.047565 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:04.047532    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"]
Nov 02 20:14:04.098677 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:04.098639    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d53422-kldrw"]
Nov 02 20:14:04.247102 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:04.247063    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:04.509219 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:04.509139    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb" podUID="20d012b3-b7bc-45da-99c6-741d3d2c01cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:14:05.168232 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:05.168200    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6aa83bd-ffa5-4346-b9b6-2ae139b9388b" path="/var/lib/kubelet/pods/a6aa83bd-ffa5-4346-b9b6-2ae139b9388b/volumes"
Nov 02 20:14:05.168620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:05.168566    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"
Nov 02 20:14:08.399603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:08.399573    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:14:09.697624 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:09.697585    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:14.509825 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:14.509754    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb" podUID="20d012b3-b7bc-45da-99c6-741d3d2c01cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:14:14.706081 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:14.706045    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:16.737261 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:16.737226    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:19.275826 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.275793    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"
Nov 02 20:14:19.328546 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.328515    5224 generic.go:356] "Generic (PLEG): container finished" podID="20d012b3-b7bc-45da-99c6-741d3d2c01cd" containerID="552d93288cc854de6ead763a01924f1b6331865f74910f7f1205f25f54e60f54" exitCode=0
Nov 02 20:14:19.328546 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.328560    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"
Nov 02 20:14:19.328757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.328600    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb" event={"ID":"20d012b3-b7bc-45da-99c6-741d3d2c01cd","Type":"ContainerDied","Data":"552d93288cc854de6ead763a01924f1b6331865f74910f7f1205f25f54e60f54"}
Nov 02 20:14:19.328757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.328652    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb" event={"ID":"20d012b3-b7bc-45da-99c6-741d3d2c01cd","Type":"ContainerDied","Data":"2b06d021c0590b481c898d3f58606a0ecdd3701951fcbf1b42355632544f02fe"}
Nov 02 20:14:19.328757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.328680    5224 scope.go:122] "RemoveContainer" containerID="552d93288cc854de6ead763a01924f1b6331865f74910f7f1205f25f54e60f54"
Nov 02 20:14:19.339213 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.339193    5224 scope.go:122] "RemoveContainer" containerID="552d93288cc854de6ead763a01924f1b6331865f74910f7f1205f25f54e60f54"
Nov 02 20:14:19.339549 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:14:19.339519    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552d93288cc854de6ead763a01924f1b6331865f74910f7f1205f25f54e60f54\": container with ID starting with 552d93288cc854de6ead763a01924f1b6331865f74910f7f1205f25f54e60f54 not found: ID does not exist" containerID="552d93288cc854de6ead763a01924f1b6331865f74910f7f1205f25f54e60f54"
Nov 02 20:14:19.339660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.339547    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552d93288cc854de6ead763a01924f1b6331865f74910f7f1205f25f54e60f54"} err="failed to get container status \"552d93288cc854de6ead763a01924f1b6331865f74910f7f1205f25f54e60f54\": rpc error: code = NotFound desc = could not find container \"552d93288cc854de6ead763a01924f1b6331865f74910f7f1205f25f54e60f54\": container with ID starting with 552d93288cc854de6ead763a01924f1b6331865f74910f7f1205f25f54e60f54 not found: ID does not exist"
Nov 02 20:14:19.424554 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.424517    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d012b3-b7bc-45da-99c6-741d3d2c01cd-kube-api-access-psn49\" (UniqueName: \"kubernetes.io/projected/20d012b3-b7bc-45da-99c6-741d3d2c01cd-kube-api-access-psn49\") pod \"20d012b3-b7bc-45da-99c6-741d3d2c01cd\" (UID: \"20d012b3-b7bc-45da-99c6-741d3d2c01cd\") "
Nov 02 20:14:19.424815 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.424793    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d012b3-b7bc-45da-99c6-741d3d2c01cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d012b3-b7bc-45da-99c6-741d3d2c01cd-plugins-config-volume\") pod \"20d012b3-b7bc-45da-99c6-741d3d2c01cd\" (UID: \"20d012b3-b7bc-45da-99c6-741d3d2c01cd\") "
Nov 02 20:14:19.425424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.425197    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d012b3-b7bc-45da-99c6-741d3d2c01cd-plugins-config-volume" pod "20d012b3-b7bc-45da-99c6-741d3d2c01cd" (UID: "20d012b3-b7bc-45da-99c6-741d3d2c01cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:19.425424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.425391    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d012b3-b7bc-45da-99c6-741d3d2c01cd-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:19.427559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.427516    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d012b3-b7bc-45da-99c6-741d3d2c01cd-kube-api-access-psn49" pod "20d012b3-b7bc-45da-99c6-741d3d2c01cd" (UID: "20d012b3-b7bc-45da-99c6-741d3d2c01cd"). InnerVolumeSpecName "kube-api-access-psn49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:19.526512 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.526481    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psn49\" (UniqueName: \"kubernetes.io/projected/20d012b3-b7bc-45da-99c6-741d3d2c01cd-kube-api-access-psn49\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:19.529785 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.529736    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:19.753171 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.753139    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"]
Nov 02 20:14:19.756391 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:19.756362    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-b98bb"]
Nov 02 20:14:21.168534 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:21.168500    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d012b3-b7bc-45da-99c6-741d3d2c01cd" path="/var/lib/kubelet/pods/20d012b3-b7bc-45da-99c6-741d3d2c01cd/volumes"
Nov 02 20:14:23.048058 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:23.048024    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"]
Nov 02 20:14:23.099147 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:23.099116    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7893d5df-52znw"]
Nov 02 20:14:23.169044 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:23.169002    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8d969f2-40af-4308-9062-f27f9063bc5f" path="/var/lib/kubelet/pods/d8d969f2-40af-4308-9062-f27f9063bc5f/volumes"
Nov 02 20:14:24.683606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:24.683571    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:27.362254 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:14:27.362223    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:14:29.215427 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:29.215389    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:30.126988 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:30.126943    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:31.151535 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:31.151502    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:35.048436 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:35.048400    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"]
Nov 02 20:14:35.099311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:35.099272    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0739db3a-t7z9m"]
Nov 02 20:14:35.168362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:35.168334    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="349a7a77-2ded-45b4-932f-493758b431a5" path="/var/lib/kubelet/pods/349a7a77-2ded-45b4-932f-493758b431a5/volumes"
Nov 02 20:14:35.467171 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:35.467076    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:36.700799 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:36.700738    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:36.880746 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:36.880713    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"]
Nov 02 20:14:36.993369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:36.993281    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"]
Nov 02 20:14:37.268634 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:37.268595    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:37.484924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:37.484892    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:38.345339 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:38.345302    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:39.528424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:39.528391    5224 scope.go:122] "RemoveContainer" containerID="4fb4ae8c2d23740b115eb41906a8d720c6acac8a1f2647afc391fd506e8611b7"
Nov 02 20:14:39.551449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:39.551417    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:40.967582 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:40.967543    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:41.881445 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:41.881383    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc" podUID="17665ced-46f5-4128-bcf5-0779a5562797" containerName="payload-processing" containerID="cri-o://e98a881dc2f7ffed2255f4d3b3e1775cf83344b73d71be70fe7269d18e2e40ba" gracePeriod=25
Nov 02 20:14:41.994371 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:41.994323    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97" podUID="d26c20ab-f400-439e-8f58-883b3fe49d2a" containerName="payload-pre-processing" containerID="cri-o://150e7b09bad7e0bb22d21a438998be41c6677203ae4de090e9f983fb07be6dfa" gracePeriod=25
Nov 02 20:14:42.148572 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.148540    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:42.161834 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.161805    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"
Nov 02 20:14:42.241821 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.241786    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17665ced-46f5-4128-bcf5-0779a5562797-kube-api-access-z5v5n\" (UniqueName: \"kubernetes.io/projected/17665ced-46f5-4128-bcf5-0779a5562797-kube-api-access-z5v5n\") pod \"17665ced-46f5-4128-bcf5-0779a5562797\" (UID: \"17665ced-46f5-4128-bcf5-0779a5562797\") "
Nov 02 20:14:42.242079 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.242060    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17665ced-46f5-4128-bcf5-0779a5562797-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17665ced-46f5-4128-bcf5-0779a5562797-plugins-config-volume\") pod \"17665ced-46f5-4128-bcf5-0779a5562797\" (UID: \"17665ced-46f5-4128-bcf5-0779a5562797\") "
Nov 02 20:14:42.243115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.242498    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17665ced-46f5-4128-bcf5-0779a5562797-plugins-config-volume" pod "17665ced-46f5-4128-bcf5-0779a5562797" (UID: "17665ced-46f5-4128-bcf5-0779a5562797"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:42.243115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.242962    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17665ced-46f5-4128-bcf5-0779a5562797-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:42.244829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.244807    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17665ced-46f5-4128-bcf5-0779a5562797-kube-api-access-z5v5n" pod "17665ced-46f5-4128-bcf5-0779a5562797" (UID: "17665ced-46f5-4128-bcf5-0779a5562797"). InnerVolumeSpecName "kube-api-access-z5v5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:42.265522 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.265499    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"
Nov 02 20:14:42.344007 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.343976    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26c20ab-f400-439e-8f58-883b3fe49d2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26c20ab-f400-439e-8f58-883b3fe49d2a-plugins-config-volume\") pod \"d26c20ab-f400-439e-8f58-883b3fe49d2a\" (UID: \"d26c20ab-f400-439e-8f58-883b3fe49d2a\") "
Nov 02 20:14:42.344195 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.344028    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26c20ab-f400-439e-8f58-883b3fe49d2a-kube-api-access-8jrgt\" (UniqueName: \"kubernetes.io/projected/d26c20ab-f400-439e-8f58-883b3fe49d2a-kube-api-access-8jrgt\") pod \"d26c20ab-f400-439e-8f58-883b3fe49d2a\" (UID: \"d26c20ab-f400-439e-8f58-883b3fe49d2a\") "
Nov 02 20:14:42.344421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.344396    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26c20ab-f400-439e-8f58-883b3fe49d2a-plugins-config-volume" pod "d26c20ab-f400-439e-8f58-883b3fe49d2a" (UID: "d26c20ab-f400-439e-8f58-883b3fe49d2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:42.344494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.344471    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5v5n\" (UniqueName: \"kubernetes.io/projected/17665ced-46f5-4128-bcf5-0779a5562797-kube-api-access-z5v5n\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:42.346109 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.346083    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26c20ab-f400-439e-8f58-883b3fe49d2a-kube-api-access-8jrgt" pod "d26c20ab-f400-439e-8f58-883b3fe49d2a" (UID: "d26c20ab-f400-439e-8f58-883b3fe49d2a"). InnerVolumeSpecName "kube-api-access-8jrgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:42.446228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.446137    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26c20ab-f400-439e-8f58-883b3fe49d2a-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:42.446228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.446179    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jrgt\" (UniqueName: \"kubernetes.io/projected/d26c20ab-f400-439e-8f58-883b3fe49d2a-kube-api-access-8jrgt\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:42.496626 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.496590    5224 generic.go:356] "Generic (PLEG): container finished" podID="17665ced-46f5-4128-bcf5-0779a5562797" containerID="e98a881dc2f7ffed2255f4d3b3e1775cf83344b73d71be70fe7269d18e2e40ba" exitCode=0
Nov 02 20:14:42.496876 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.496639    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"
Nov 02 20:14:42.496876 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.496678    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc" event={"ID":"17665ced-46f5-4128-bcf5-0779a5562797","Type":"ContainerDied","Data":"e98a881dc2f7ffed2255f4d3b3e1775cf83344b73d71be70fe7269d18e2e40ba"}
Nov 02 20:14:42.496876 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.496746    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc" event={"ID":"17665ced-46f5-4128-bcf5-0779a5562797","Type":"ContainerDied","Data":"34d0d6fb2169f70b1b20e8ece024d51731fccc7df3a5b8021302d632147334db"}
Nov 02 20:14:42.496876 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.496792    5224 scope.go:122] "RemoveContainer" containerID="e98a881dc2f7ffed2255f4d3b3e1775cf83344b73d71be70fe7269d18e2e40ba"
Nov 02 20:14:42.498456 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.498427    5224 generic.go:356] "Generic (PLEG): container finished" podID="d26c20ab-f400-439e-8f58-883b3fe49d2a" containerID="150e7b09bad7e0bb22d21a438998be41c6677203ae4de090e9f983fb07be6dfa" exitCode=0
Nov 02 20:14:42.498581 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.498464    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"
Nov 02 20:14:42.498581 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.498476    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97" event={"ID":"d26c20ab-f400-439e-8f58-883b3fe49d2a","Type":"ContainerDied","Data":"150e7b09bad7e0bb22d21a438998be41c6677203ae4de090e9f983fb07be6dfa"}
Nov 02 20:14:42.498581 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.498527    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97" event={"ID":"d26c20ab-f400-439e-8f58-883b3fe49d2a","Type":"ContainerDied","Data":"8a39676d84135e7ef656ffb2e883cefcbd72e23f4f6d7dd1f469d03bd7bd8d30"}
Nov 02 20:14:42.508042 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.508017    5224 scope.go:122] "RemoveContainer" containerID="e98a881dc2f7ffed2255f4d3b3e1775cf83344b73d71be70fe7269d18e2e40ba"
Nov 02 20:14:42.508367 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:14:42.508345    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98a881dc2f7ffed2255f4d3b3e1775cf83344b73d71be70fe7269d18e2e40ba\": container with ID starting with e98a881dc2f7ffed2255f4d3b3e1775cf83344b73d71be70fe7269d18e2e40ba not found: ID does not exist" containerID="e98a881dc2f7ffed2255f4d3b3e1775cf83344b73d71be70fe7269d18e2e40ba"
Nov 02 20:14:42.508464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.508375    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98a881dc2f7ffed2255f4d3b3e1775cf83344b73d71be70fe7269d18e2e40ba"} err="failed to get container status \"e98a881dc2f7ffed2255f4d3b3e1775cf83344b73d71be70fe7269d18e2e40ba\": rpc error: code = NotFound desc = could not find container \"e98a881dc2f7ffed2255f4d3b3e1775cf83344b73d71be70fe7269d18e2e40ba\": container with ID starting with e98a881dc2f7ffed2255f4d3b3e1775cf83344b73d71be70fe7269d18e2e40ba not found: ID does not exist"
Nov 02 20:14:42.508464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.508393    5224 scope.go:122] "RemoveContainer" containerID="150e7b09bad7e0bb22d21a438998be41c6677203ae4de090e9f983fb07be6dfa"
Nov 02 20:14:42.517683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.517664    5224 scope.go:122] "RemoveContainer" containerID="150e7b09bad7e0bb22d21a438998be41c6677203ae4de090e9f983fb07be6dfa"
Nov 02 20:14:42.518033 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:14:42.518015    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150e7b09bad7e0bb22d21a438998be41c6677203ae4de090e9f983fb07be6dfa\": container with ID starting with 150e7b09bad7e0bb22d21a438998be41c6677203ae4de090e9f983fb07be6dfa not found: ID does not exist" containerID="150e7b09bad7e0bb22d21a438998be41c6677203ae4de090e9f983fb07be6dfa"
Nov 02 20:14:42.518085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.518041    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150e7b09bad7e0bb22d21a438998be41c6677203ae4de090e9f983fb07be6dfa"} err="failed to get container status \"150e7b09bad7e0bb22d21a438998be41c6677203ae4de090e9f983fb07be6dfa\": rpc error: code = NotFound desc = could not find container \"150e7b09bad7e0bb22d21a438998be41c6677203ae4de090e9f983fb07be6dfa\": container with ID starting with 150e7b09bad7e0bb22d21a438998be41c6677203ae4de090e9f983fb07be6dfa not found: ID does not exist"
Nov 02 20:14:42.665093 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.665061    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"]
Nov 02 20:14:42.669307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.669282    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45c182-7d49d6fcb7-h2v97"]
Nov 02 20:14:42.779446 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.779413    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"]
Nov 02 20:14:42.782417 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:42.782394    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-45c182-6999b5dcd5-627jc"]
Nov 02 20:14:43.168398 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:43.168325    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17665ced-46f5-4128-bcf5-0779a5562797" path="/var/lib/kubelet/pods/17665ced-46f5-4128-bcf5-0779a5562797/volumes"
Nov 02 20:14:43.168800 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:43.168783    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26c20ab-f400-439e-8f58-883b3fe49d2a" path="/var/lib/kubelet/pods/d26c20ab-f400-439e-8f58-883b3fe49d2a/volumes"
Nov 02 20:14:45.211419 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:45.211380    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:46.001657 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:46.001620    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:46.168655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:46.168619    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:46.339730 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:46.339697    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:48.227649 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:48.227617    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"]
Nov 02 20:14:48.335085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:48.335049    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"]
Nov 02 20:14:48.610177 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:48.610142    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:48.830276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:48.830238    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:49.059049 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:49.058993    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"]
Nov 02 20:14:49.060166 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:49.060140    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f58366-m9wn5"]
Nov 02 20:14:49.168440 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:49.168410    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef576c8-8431-42a0-831b-1d87de8d8ed6" path="/var/lib/kubelet/pods/9ef576c8-8431-42a0-831b-1d87de8d8ed6/volumes"
Nov 02 20:14:50.926139 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:50.926105    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:51.529473 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:51.529441    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:14:51.582341 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:51.582312    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:53.228126 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.228077    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx" podUID="3bc4e74f-60ce-4443-a161-505979d549e7" containerName="payload-processing" containerID="cri-o://1fa8ff778602471ab961dbd8f184e663d323ec1f5d3c0a4efecf95ad06d75e0f" gracePeriod=25
Nov 02 20:14:53.336384 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.336342    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx" podUID="73c7a129-8af4-41d3-bcd5-84c070f1843e" containerName="payload-pre-processing" containerID="cri-o://98bf424aadc7f54aede80818285a0f60b65d7e6c9901744b4a2cc126a2808e5e" gracePeriod=25
Nov 02 20:14:53.465934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.465897    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:53.500845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.500810    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"
Nov 02 20:14:53.612128 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.612096    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc4e74f-60ce-4443-a161-505979d549e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc4e74f-60ce-4443-a161-505979d549e7-plugins-config-volume\") pod \"3bc4e74f-60ce-4443-a161-505979d549e7\" (UID: \"3bc4e74f-60ce-4443-a161-505979d549e7\") "
Nov 02 20:14:53.612530 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.612371    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc4e74f-60ce-4443-a161-505979d549e7-kube-api-access-mm8r5\" (UniqueName: \"kubernetes.io/projected/3bc4e74f-60ce-4443-a161-505979d549e7-kube-api-access-mm8r5\") pod \"3bc4e74f-60ce-4443-a161-505979d549e7\" (UID: \"3bc4e74f-60ce-4443-a161-505979d549e7\") "
Nov 02 20:14:53.613325 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.613282    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc4e74f-60ce-4443-a161-505979d549e7-plugins-config-volume" pod "3bc4e74f-60ce-4443-a161-505979d549e7" (UID: "3bc4e74f-60ce-4443-a161-505979d549e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:53.615003 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.614691    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc4e74f-60ce-4443-a161-505979d549e7-kube-api-access-mm8r5" pod "3bc4e74f-60ce-4443-a161-505979d549e7" (UID: "3bc4e74f-60ce-4443-a161-505979d549e7"). InnerVolumeSpecName "kube-api-access-mm8r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:53.619179 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.619149    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"
Nov 02 20:14:53.623569 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.623542    5224 generic.go:356] "Generic (PLEG): container finished" podID="73c7a129-8af4-41d3-bcd5-84c070f1843e" containerID="98bf424aadc7f54aede80818285a0f60b65d7e6c9901744b4a2cc126a2808e5e" exitCode=0
Nov 02 20:14:53.623669 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.623634    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"
Nov 02 20:14:53.623720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.623671    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx" event={"ID":"73c7a129-8af4-41d3-bcd5-84c070f1843e","Type":"ContainerDied","Data":"98bf424aadc7f54aede80818285a0f60b65d7e6c9901744b4a2cc126a2808e5e"}
Nov 02 20:14:53.623794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.623719    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx" event={"ID":"73c7a129-8af4-41d3-bcd5-84c070f1843e","Type":"ContainerDied","Data":"4176db5a2d48c6b701e19700491bcc89ad0582b10619a150c01a1ea36aace536"}
Nov 02 20:14:53.623794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.623741    5224 scope.go:122] "RemoveContainer" containerID="98bf424aadc7f54aede80818285a0f60b65d7e6c9901744b4a2cc126a2808e5e"
Nov 02 20:14:53.625436 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.625375    5224 generic.go:356] "Generic (PLEG): container finished" podID="3bc4e74f-60ce-4443-a161-505979d549e7" containerID="1fa8ff778602471ab961dbd8f184e663d323ec1f5d3c0a4efecf95ad06d75e0f" exitCode=0
Nov 02 20:14:53.625436 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.625421    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"
Nov 02 20:14:53.625644 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.625416    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx" event={"ID":"3bc4e74f-60ce-4443-a161-505979d549e7","Type":"ContainerDied","Data":"1fa8ff778602471ab961dbd8f184e663d323ec1f5d3c0a4efecf95ad06d75e0f"}
Nov 02 20:14:53.625644 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.625522    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx" event={"ID":"3bc4e74f-60ce-4443-a161-505979d549e7","Type":"ContainerDied","Data":"89cd8d70661b52e56100f45a9114f0b9fa258d0c029255b7155e3589683abe01"}
Nov 02 20:14:53.637357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.637332    5224 scope.go:122] "RemoveContainer" containerID="98bf424aadc7f54aede80818285a0f60b65d7e6c9901744b4a2cc126a2808e5e"
Nov 02 20:14:53.637646 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:14:53.637626    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98bf424aadc7f54aede80818285a0f60b65d7e6c9901744b4a2cc126a2808e5e\": container with ID starting with 98bf424aadc7f54aede80818285a0f60b65d7e6c9901744b4a2cc126a2808e5e not found: ID does not exist" containerID="98bf424aadc7f54aede80818285a0f60b65d7e6c9901744b4a2cc126a2808e5e"
Nov 02 20:14:53.637730 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.637654    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98bf424aadc7f54aede80818285a0f60b65d7e6c9901744b4a2cc126a2808e5e"} err="failed to get container status \"98bf424aadc7f54aede80818285a0f60b65d7e6c9901744b4a2cc126a2808e5e\": rpc error: code = NotFound desc = could not find container \"98bf424aadc7f54aede80818285a0f60b65d7e6c9901744b4a2cc126a2808e5e\": container with ID starting with 98bf424aadc7f54aede80818285a0f60b65d7e6c9901744b4a2cc126a2808e5e not found: ID does not exist"
Nov 02 20:14:53.637730 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.637672    5224 scope.go:122] "RemoveContainer" containerID="1fa8ff778602471ab961dbd8f184e663d323ec1f5d3c0a4efecf95ad06d75e0f"
Nov 02 20:14:53.648746 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.648730    5224 scope.go:122] "RemoveContainer" containerID="1fa8ff778602471ab961dbd8f184e663d323ec1f5d3c0a4efecf95ad06d75e0f"
Nov 02 20:14:53.649102 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:14:53.649079    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa8ff778602471ab961dbd8f184e663d323ec1f5d3c0a4efecf95ad06d75e0f\": container with ID starting with 1fa8ff778602471ab961dbd8f184e663d323ec1f5d3c0a4efecf95ad06d75e0f not found: ID does not exist" containerID="1fa8ff778602471ab961dbd8f184e663d323ec1f5d3c0a4efecf95ad06d75e0f"
Nov 02 20:14:53.649158 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.649110    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa8ff778602471ab961dbd8f184e663d323ec1f5d3c0a4efecf95ad06d75e0f"} err="failed to get container status \"1fa8ff778602471ab961dbd8f184e663d323ec1f5d3c0a4efecf95ad06d75e0f\": rpc error: code = NotFound desc = could not find container \"1fa8ff778602471ab961dbd8f184e663d323ec1f5d3c0a4efecf95ad06d75e0f\": container with ID starting with 1fa8ff778602471ab961dbd8f184e663d323ec1f5d3c0a4efecf95ad06d75e0f not found: ID does not exist"
Nov 02 20:14:53.714029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.713984    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c7a129-8af4-41d3-bcd5-84c070f1843e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c7a129-8af4-41d3-bcd5-84c070f1843e-plugins-config-volume\") pod \"73c7a129-8af4-41d3-bcd5-84c070f1843e\" (UID: \"73c7a129-8af4-41d3-bcd5-84c070f1843e\") "
Nov 02 20:14:53.714237 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.714146    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c7a129-8af4-41d3-bcd5-84c070f1843e-kube-api-access-67qzv\" (UniqueName: \"kubernetes.io/projected/73c7a129-8af4-41d3-bcd5-84c070f1843e-kube-api-access-67qzv\") pod \"73c7a129-8af4-41d3-bcd5-84c070f1843e\" (UID: \"73c7a129-8af4-41d3-bcd5-84c070f1843e\") "
Nov 02 20:14:53.714440 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.714414    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c7a129-8af4-41d3-bcd5-84c070f1843e-plugins-config-volume" pod "73c7a129-8af4-41d3-bcd5-84c070f1843e" (UID: "73c7a129-8af4-41d3-bcd5-84c070f1843e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:53.714616 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.714598    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc4e74f-60ce-4443-a161-505979d549e7-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:53.714671 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.714622    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c7a129-8af4-41d3-bcd5-84c070f1843e-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:53.714671 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.714639    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm8r5\" (UniqueName: \"kubernetes.io/projected/3bc4e74f-60ce-4443-a161-505979d549e7-kube-api-access-mm8r5\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:53.716247 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.716221    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c7a129-8af4-41d3-bcd5-84c070f1843e-kube-api-access-67qzv" pod "73c7a129-8af4-41d3-bcd5-84c070f1843e" (UID: "73c7a129-8af4-41d3-bcd5-84c070f1843e"). InnerVolumeSpecName "kube-api-access-67qzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:53.815963 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.815929    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67qzv\" (UniqueName: \"kubernetes.io/projected/73c7a129-8af4-41d3-bcd5-84c070f1843e-kube-api-access-67qzv\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:53.904129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.904086    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"]
Nov 02 20:14:53.907109 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:53.907078    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08eeaa-668dc5d9d7-brmpx"]
Nov 02 20:14:54.047409 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:54.047373    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"]
Nov 02 20:14:54.053006 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:54.052981    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08eeaa-6cbf9f5c47-pkdbx"]
Nov 02 20:14:54.311131 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:54.311099    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:55.168387 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:55.168346    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc4e74f-60ce-4443-a161-505979d549e7" path="/var/lib/kubelet/pods/3bc4e74f-60ce-4443-a161-505979d549e7/volumes"
Nov 02 20:14:55.168903 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:55.168882    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c7a129-8af4-41d3-bcd5-84c070f1843e" path="/var/lib/kubelet/pods/73c7a129-8af4-41d3-bcd5-84c070f1843e/volumes"
Nov 02 20:14:57.188111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:14:57.184804    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:14:57.359841 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:14:57.359805    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:15:00.162695 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.162661    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"]
Nov 02 20:15:00.163496 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163472    5224 cpu_manager.go:414] "containerMap: removing container" podUID="73c7a129-8af4-41d3-bcd5-84c070f1843e" containerName="payload-pre-processing"
Nov 02 20:15:00.163598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163499    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c7a129-8af4-41d3-bcd5-84c070f1843e" containerName="payload-pre-processing"
Nov 02 20:15:00.163598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163534    5224 cpu_manager.go:414] "containerMap: removing container" podUID="20d012b3-b7bc-45da-99c6-741d3d2c01cd" containerName="payload-processing"
Nov 02 20:15:00.163598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163542    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d012b3-b7bc-45da-99c6-741d3d2c01cd" containerName="payload-processing"
Nov 02 20:15:00.163598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163561    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d26c20ab-f400-439e-8f58-883b3fe49d2a" containerName="payload-pre-processing"
Nov 02 20:15:00.163598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163570    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26c20ab-f400-439e-8f58-883b3fe49d2a" containerName="payload-pre-processing"
Nov 02 20:15:00.163598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163585    5224 cpu_manager.go:414] "containerMap: removing container" podUID="17665ced-46f5-4128-bcf5-0779a5562797" containerName="payload-processing"
Nov 02 20:15:00.163598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163593    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17665ced-46f5-4128-bcf5-0779a5562797" containerName="payload-processing"
Nov 02 20:15:00.163938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163624    5224 cpu_manager.go:414] "containerMap: removing container" podUID="3bc4e74f-60ce-4443-a161-505979d549e7" containerName="payload-processing"
Nov 02 20:15:00.163938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163631    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc4e74f-60ce-4443-a161-505979d549e7" containerName="payload-processing"
Nov 02 20:15:00.163938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163807    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d012b3-b7bc-45da-99c6-741d3d2c01cd" containerName="payload-processing"
Nov 02 20:15:00.163938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163824    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26c20ab-f400-439e-8f58-883b3fe49d2a" containerName="payload-pre-processing"
Nov 02 20:15:00.163938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163834    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="17665ced-46f5-4128-bcf5-0779a5562797" containerName="payload-processing"
Nov 02 20:15:00.163938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163848    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc4e74f-60ce-4443-a161-505979d549e7" containerName="payload-processing"
Nov 02 20:15:00.163938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.163862    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c7a129-8af4-41d3-bcd5-84c070f1843e" containerName="payload-pre-processing"
Nov 02 20:15:00.170063 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.168155    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"]
Nov 02 20:15:00.173559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.173529    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"
Nov 02 20:15:00.174783 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.174754    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"]
Nov 02 20:15:00.175345 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.175325    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"
Nov 02 20:15:00.180411 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.180388    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"
Nov 02 20:15:00.213851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.213824    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"]
Nov 02 20:15:00.218473 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.218454    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"]
Nov 02 20:15:00.218569 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.218553    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"
Nov 02 20:15:00.223278 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.223260    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"]
Nov 02 20:15:00.223379 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.223359    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"
Nov 02 20:15:00.227830 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.227815    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"]
Nov 02 20:15:00.227968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.227946    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"
Nov 02 20:15:00.231884 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.231866    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"
Nov 02 20:15:00.253191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.253168    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"]
Nov 02 20:15:00.265202 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.265172    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"]
Nov 02 20:15:00.268454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.268432    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"]
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.301127    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"]
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302013    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2189a356-2417-4635-b1b4-22f3920a5f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx\" (UID: \"2189a356-2417-4635-b1b4-22f3920a5f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302055    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f31166-a27b-4a71-98d0-a727faea4589-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb\" (UID: \"95f31166-a27b-4a71-98d0-a727faea4589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302114    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq\" (UID: \"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302139    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8slff\" (UniqueName: \"kubernetes.io/projected/2189a356-2417-4635-b1b4-22f3920a5f21-kube-api-access-8slff\") pod \"maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx\" (UID: \"2189a356-2417-4635-b1b4-22f3920a5f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302168    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqxbd\" (UniqueName: \"kubernetes.io/projected/95f31166-a27b-4a71-98d0-a727faea4589-kube-api-access-rqxbd\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb\" (UID: \"95f31166-a27b-4a71-98d0-a727faea4589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302199    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b298bc-073a-49e8-86ae-998b10102bfe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8\" (UID: \"56b298bc-073a-49e8-86ae-998b10102bfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302306    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8s75\" (UniqueName: \"kubernetes.io/projected/ed59494c-0a34-41ab-ba57-ae829d35f753-kube-api-access-q8s75\") pod \"maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm\" (UID: \"ed59494c-0a34-41ab-ba57-ae829d35f753\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302335    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmk98\" (UniqueName: \"kubernetes.io/projected/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-kube-api-access-vmk98\") pod \"maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq\" (UID: \"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302414    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq6r5\" (UniqueName: \"kubernetes.io/projected/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-kube-api-access-qq6r5\") pod \"maas-api-key-cleanup-29894175-9zws2\" (UID: \"dbf16df4-48e0-417b-8a03-82cf8fb4a10a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302450    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed59494c-0a34-41ab-ba57-ae829d35f753-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm\" (UID: \"ed59494c-0a34-41ab-ba57-ae829d35f753\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302510    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63752026-3fce-43fd-8eb5-b0dfb4e4c398-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f\" (UID: \"63752026-3fce-43fd-8eb5-b0dfb4e4c398\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302560    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894175-9zws2\" (UID: \"dbf16df4-48e0-417b-8a03-82cf8fb4a10a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302626    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkkjs\" (UniqueName: \"kubernetes.io/projected/63752026-3fce-43fd-8eb5-b0dfb4e4c398-kube-api-access-fkkjs\") pod \"maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f\" (UID: \"63752026-3fce-43fd-8eb5-b0dfb4e4c398\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"
Nov 02 20:15:00.303168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.302661    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k2wp\" (UniqueName: \"kubernetes.io/projected/56b298bc-073a-49e8-86ae-998b10102bfe-kube-api-access-4k2wp\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8\" (UID: \"56b298bc-073a-49e8-86ae-998b10102bfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"
Nov 02 20:15:00.306538 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.306516    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"]
Nov 02 20:15:00.309421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.309400    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"]
Nov 02 20:15:00.321319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.321289    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:15:00.321319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.321300    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:15:00.321445 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.321328    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:15:00.337815 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.336500    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"]
Nov 02 20:15:00.403844 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.403812    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2189a356-2417-4635-b1b4-22f3920a5f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx\" (UID: \"2189a356-2417-4635-b1b4-22f3920a5f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"
Nov 02 20:15:00.404021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.403860    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f31166-a27b-4a71-98d0-a727faea4589-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb\" (UID: \"95f31166-a27b-4a71-98d0-a727faea4589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"
Nov 02 20:15:00.404021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.403913    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq\" (UID: \"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"
Nov 02 20:15:00.404021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.403936    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8slff\" (UniqueName: \"kubernetes.io/projected/2189a356-2417-4635-b1b4-22f3920a5f21-kube-api-access-8slff\") pod \"maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx\" (UID: \"2189a356-2417-4635-b1b4-22f3920a5f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"
Nov 02 20:15:00.404021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.403966    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqxbd\" (UniqueName: \"kubernetes.io/projected/95f31166-a27b-4a71-98d0-a727faea4589-kube-api-access-rqxbd\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb\" (UID: \"95f31166-a27b-4a71-98d0-a727faea4589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"
Nov 02 20:15:00.404021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.403995    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b298bc-073a-49e8-86ae-998b10102bfe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8\" (UID: \"56b298bc-073a-49e8-86ae-998b10102bfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"
Nov 02 20:15:00.404283 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.404110    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8s75\" (UniqueName: \"kubernetes.io/projected/ed59494c-0a34-41ab-ba57-ae829d35f753-kube-api-access-q8s75\") pod \"maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm\" (UID: \"ed59494c-0a34-41ab-ba57-ae829d35f753\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"
Nov 02 20:15:00.404283 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.404145    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmk98\" (UniqueName: \"kubernetes.io/projected/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-kube-api-access-vmk98\") pod \"maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq\" (UID: \"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"
Nov 02 20:15:00.404283 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.404229    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq6r5\" (UniqueName: \"kubernetes.io/projected/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-kube-api-access-qq6r5\") pod \"maas-api-key-cleanup-29894175-9zws2\" (UID: \"dbf16df4-48e0-417b-8a03-82cf8fb4a10a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"
Nov 02 20:15:00.404283 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.404264    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed59494c-0a34-41ab-ba57-ae829d35f753-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm\" (UID: \"ed59494c-0a34-41ab-ba57-ae829d35f753\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"
Nov 02 20:15:00.404517 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.404329    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63752026-3fce-43fd-8eb5-b0dfb4e4c398-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f\" (UID: \"63752026-3fce-43fd-8eb5-b0dfb4e4c398\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"
Nov 02 20:15:00.404517 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.404385    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894175-9zws2\" (UID: \"dbf16df4-48e0-417b-8a03-82cf8fb4a10a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"
Nov 02 20:15:00.404517 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.404452    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkkjs\" (UniqueName: \"kubernetes.io/projected/63752026-3fce-43fd-8eb5-b0dfb4e4c398-kube-api-access-fkkjs\") pod \"maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f\" (UID: \"63752026-3fce-43fd-8eb5-b0dfb4e4c398\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"
Nov 02 20:15:00.404517 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.404485    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k2wp\" (UniqueName: \"kubernetes.io/projected/56b298bc-073a-49e8-86ae-998b10102bfe-kube-api-access-4k2wp\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8\" (UID: \"56b298bc-073a-49e8-86ae-998b10102bfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"
Nov 02 20:15:00.404720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.404697    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2189a356-2417-4635-b1b4-22f3920a5f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx\" (UID: \"2189a356-2417-4635-b1b4-22f3920a5f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"
Nov 02 20:15:00.404720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.404711    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq\" (UID: \"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"
Nov 02 20:15:00.404858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.404736    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f31166-a27b-4a71-98d0-a727faea4589-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb\" (UID: \"95f31166-a27b-4a71-98d0-a727faea4589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"
Nov 02 20:15:00.404914 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.404897    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b298bc-073a-49e8-86ae-998b10102bfe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8\" (UID: \"56b298bc-073a-49e8-86ae-998b10102bfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"
Nov 02 20:15:00.405179 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.405159    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63752026-3fce-43fd-8eb5-b0dfb4e4c398-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f\" (UID: \"63752026-3fce-43fd-8eb5-b0dfb4e4c398\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"
Nov 02 20:15:00.405213 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.405159    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed59494c-0a34-41ab-ba57-ae829d35f753-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm\" (UID: \"ed59494c-0a34-41ab-ba57-ae829d35f753\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"
Nov 02 20:15:00.405283 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.405266    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894175-9zws2\" (UID: \"dbf16df4-48e0-417b-8a03-82cf8fb4a10a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"
Nov 02 20:15:00.463266 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.463234    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k2wp\" (UniqueName: \"kubernetes.io/projected/56b298bc-073a-49e8-86ae-998b10102bfe-kube-api-access-4k2wp\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8\" (UID: \"56b298bc-073a-49e8-86ae-998b10102bfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"
Nov 02 20:15:00.463412 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.463292    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkkjs\" (UniqueName: \"kubernetes.io/projected/63752026-3fce-43fd-8eb5-b0dfb4e4c398-kube-api-access-fkkjs\") pod \"maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f\" (UID: \"63752026-3fce-43fd-8eb5-b0dfb4e4c398\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"
Nov 02 20:15:00.463572 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.463550    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8slff\" (UniqueName: \"kubernetes.io/projected/2189a356-2417-4635-b1b4-22f3920a5f21-kube-api-access-8slff\") pod \"maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx\" (UID: \"2189a356-2417-4635-b1b4-22f3920a5f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"
Nov 02 20:15:00.463656 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.463621    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqxbd\" (UniqueName: \"kubernetes.io/projected/95f31166-a27b-4a71-98d0-a727faea4589-kube-api-access-rqxbd\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb\" (UID: \"95f31166-a27b-4a71-98d0-a727faea4589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"
Nov 02 20:15:00.463656 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.463648    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmk98\" (UniqueName: \"kubernetes.io/projected/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-kube-api-access-vmk98\") pod \"maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq\" (UID: \"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"
Nov 02 20:15:00.464063 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.464026    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8s75\" (UniqueName: \"kubernetes.io/projected/ed59494c-0a34-41ab-ba57-ae829d35f753-kube-api-access-q8s75\") pod \"maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm\" (UID: \"ed59494c-0a34-41ab-ba57-ae829d35f753\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"
Nov 02 20:15:00.464151 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.464105    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq6r5\" (UniqueName: \"kubernetes.io/projected/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-kube-api-access-qq6r5\") pod \"maas-api-key-cleanup-29894175-9zws2\" (UID: \"dbf16df4-48e0-417b-8a03-82cf8fb4a10a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"
Nov 02 20:15:00.495594 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.495570    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"
Nov 02 20:15:00.501075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.501051    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"
Nov 02 20:15:00.506279 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.506263    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"
Nov 02 20:15:00.531647 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.531618    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"
Nov 02 20:15:00.542414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.542213    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"
Nov 02 20:15:00.550615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.550346    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"
Nov 02 20:15:00.557231 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.557058    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"
Nov 02 20:15:00.892493 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:15:00.892464    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2189a356_2417_4635_b1b4_22f3920a5f21.slice/crio-1ca314dbff88ebb8da6fabce074e058a4c0d76b28e1474fa5815a406d0f1fd5e WatchSource:0}: Error finding container 1ca314dbff88ebb8da6fabce074e058a4c0d76b28e1474fa5815a406d0f1fd5e: Status 404 returned error can't find the container with id 1ca314dbff88ebb8da6fabce074e058a4c0d76b28e1474fa5815a406d0f1fd5e
Nov 02 20:15:00.897000 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.896980    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"]
Nov 02 20:15:00.937083 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:15:00.937057    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4614db6_6d7e_4ae9_bcc7_b1d746a45b8d.slice/crio-29d738a810c1ad11bea78b50a32a30a3abc58fa9bcde6beadd0c3ee09914614a WatchSource:0}: Error finding container 29d738a810c1ad11bea78b50a32a30a3abc58fa9bcde6beadd0c3ee09914614a: Status 404 returned error can't find the container with id 29d738a810c1ad11bea78b50a32a30a3abc58fa9bcde6beadd0c3ee09914614a
Nov 02 20:15:00.938153 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:15:00.938127    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f31166_a27b_4a71_98d0_a727faea4589.slice/crio-acee708192dc99a6863ecec9dbe97c485090d2b6b71dfb072cfe7a6304dbac6e WatchSource:0}: Error finding container acee708192dc99a6863ecec9dbe97c485090d2b6b71dfb072cfe7a6304dbac6e: Status 404 returned error can't find the container with id acee708192dc99a6863ecec9dbe97c485090d2b6b71dfb072cfe7a6304dbac6e
Nov 02 20:15:00.942293 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.942272    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"]
Nov 02 20:15:00.945343 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.945319    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"]
Nov 02 20:15:00.983453 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:15:00.983427    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b298bc_073a_49e8_86ae_998b10102bfe.slice/crio-758369e3b9331f9f551d3e204abb094d2ced0bc61b1177edca1c69ea8abe2701 WatchSource:0}: Error finding container 758369e3b9331f9f551d3e204abb094d2ced0bc61b1177edca1c69ea8abe2701: Status 404 returned error can't find the container with id 758369e3b9331f9f551d3e204abb094d2ced0bc61b1177edca1c69ea8abe2701
Nov 02 20:15:00.987035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.987008    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"]
Nov 02 20:15:00.996445 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.996420    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"]
Nov 02 20:15:00.996736 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:15:00.996714    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded59494c_0a34_41ab_ba57_ae829d35f753.slice/crio-c24d4b8c708d18d59891cbe3801120631c11421ef05824e742a2c9e12e963dc6 WatchSource:0}: Error finding container c24d4b8c708d18d59891cbe3801120631c11421ef05824e742a2c9e12e963dc6: Status 404 returned error can't find the container with id c24d4b8c708d18d59891cbe3801120631c11421ef05824e742a2c9e12e963dc6
Nov 02 20:15:00.999006 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:15:00.998982    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf16df4_48e0_417b_8a03_82cf8fb4a10a.slice/crio-ed3daeddf5f63a0b0765cba82f53f2c208018faca5b6c1e14019f3b849e1d4f7 WatchSource:0}: Error finding container ed3daeddf5f63a0b0765cba82f53f2c208018faca5b6c1e14019f3b849e1d4f7: Status 404 returned error can't find the container with id ed3daeddf5f63a0b0765cba82f53f2c208018faca5b6c1e14019f3b849e1d4f7
Nov 02 20:15:00.999627 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:00.999605    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"]
Nov 02 20:15:01.035050 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:15:01.035023    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63752026_3fce_43fd_8eb5_b0dfb4e4c398.slice/crio-b3e29e75a6496f510517c535612afa3c3688420b713ad90ac5f78f4185a0d6f9 WatchSource:0}: Error finding container b3e29e75a6496f510517c535612afa3c3688420b713ad90ac5f78f4185a0d6f9: Status 404 returned error can't find the container with id b3e29e75a6496f510517c535612afa3c3688420b713ad90ac5f78f4185a0d6f9
Nov 02 20:15:01.038169 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.038143    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"]
Nov 02 20:15:01.706414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.706372    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f" event={"ID":"63752026-3fce-43fd-8eb5-b0dfb4e4c398","Type":"ContainerStarted","Data":"6a53a4766dbc48b22380636a47ff698dd776878acedc5cb98f0bd651d14b13f0"}
Nov 02 20:15:01.706902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.706438    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f" event={"ID":"63752026-3fce-43fd-8eb5-b0dfb4e4c398","Type":"ContainerStarted","Data":"b3e29e75a6496f510517c535612afa3c3688420b713ad90ac5f78f4185a0d6f9"}
Nov 02 20:15:01.708132 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.708104    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2" event={"ID":"dbf16df4-48e0-417b-8a03-82cf8fb4a10a","Type":"ContainerStarted","Data":"acee544c24adbf1cabaa8968d383cd0c9c257235d27b74edf0fff50c52fa508b"}
Nov 02 20:15:01.708272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.708156    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2" event={"ID":"dbf16df4-48e0-417b-8a03-82cf8fb4a10a","Type":"ContainerStarted","Data":"ed3daeddf5f63a0b0765cba82f53f2c208018faca5b6c1e14019f3b849e1d4f7"}
Nov 02 20:15:01.709648 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.709612    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx" event={"ID":"2189a356-2417-4635-b1b4-22f3920a5f21","Type":"ContainerStarted","Data":"d757edff1eb3fdbbe7973999a928c3824d0d6c5ff910a5880bdb9e19300b385a"}
Nov 02 20:15:01.709732 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.709662    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx" event={"ID":"2189a356-2417-4635-b1b4-22f3920a5f21","Type":"ContainerStarted","Data":"1ca314dbff88ebb8da6fabce074e058a4c0d76b28e1474fa5815a406d0f1fd5e"}
Nov 02 20:15:01.711290 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.711266    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm" event={"ID":"ed59494c-0a34-41ab-ba57-ae829d35f753","Type":"ContainerStarted","Data":"ab3b5bae6eccef31975966327310ebc0bc60a41bcd1ab67ff7526689671e2464"}
Nov 02 20:15:01.711391 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.711314    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm" event={"ID":"ed59494c-0a34-41ab-ba57-ae829d35f753","Type":"ContainerStarted","Data":"c24d4b8c708d18d59891cbe3801120631c11421ef05824e742a2c9e12e963dc6"}
Nov 02 20:15:01.712751 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.712730    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8" event={"ID":"56b298bc-073a-49e8-86ae-998b10102bfe","Type":"ContainerStarted","Data":"e783b0d575cbfe790cae80f7281ea12d38ba4be51234dfbbea6324dc169088b8"}
Nov 02 20:15:01.712852 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.712790    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8" event={"ID":"56b298bc-073a-49e8-86ae-998b10102bfe","Type":"ContainerStarted","Data":"758369e3b9331f9f551d3e204abb094d2ced0bc61b1177edca1c69ea8abe2701"}
Nov 02 20:15:01.714305 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.714285    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq" event={"ID":"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d","Type":"ContainerStarted","Data":"62f72ddabc48ab8e47c21145bab6e9f3ccdde544083ed276e80a976ccecbc077"}
Nov 02 20:15:01.714410 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.714320    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq" event={"ID":"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d","Type":"ContainerStarted","Data":"29d738a810c1ad11bea78b50a32a30a3abc58fa9bcde6beadd0c3ee09914614a"}
Nov 02 20:15:01.715811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.715761    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb" event={"ID":"95f31166-a27b-4a71-98d0-a727faea4589","Type":"ContainerStarted","Data":"bb0d4699091fe31689230919f1ef5ce39a2db435f964bd7efa69f4cc2308c0e8"}
Nov 02 20:15:01.715895 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.715830    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb" event={"ID":"95f31166-a27b-4a71-98d0-a727faea4589","Type":"ContainerStarted","Data":"acee708192dc99a6863ecec9dbe97c485090d2b6b71dfb072cfe7a6304dbac6e"}
Nov 02 20:15:01.821858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.821801    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f" podStartSLOduration=1.8217878889999999 podStartE2EDuration="1.821787889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:01.820539862 +0000 UTC m=+1582.818228471" watchObservedRunningTime="2026-11-02 20:15:01.821787889 +0000 UTC m=+1582.819476494"
Nov 02 20:15:01.937044 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:01.936995    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8" podStartSLOduration=1.936981636 podStartE2EDuration="1.936981636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:01.935997493 +0000 UTC m=+1582.933686102" watchObservedRunningTime="2026-11-02 20:15:01.936981636 +0000 UTC m=+1582.934670244"
Nov 02 20:15:02.056528 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.056470    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx" podStartSLOduration=2.056450176 podStartE2EDuration="2.056450176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.05256421 +0000 UTC m=+1583.050252844" watchObservedRunningTime="2026-11-02 20:15:02.056450176 +0000 UTC m=+1583.054138785"
Nov 02 20:15:02.171836 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.171784    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm" podStartSLOduration=2.171753355 podStartE2EDuration="2.171753355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.169619397 +0000 UTC m=+1583.167308005" watchObservedRunningTime="2026-11-02 20:15:02.171753355 +0000 UTC m=+1583.169442001"
Nov 02 20:15:02.288075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.288018    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq" podStartSLOduration=2.288003634 podStartE2EDuration="2.288003634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.284054491 +0000 UTC m=+1583.281743100" watchObservedRunningTime="2026-11-02 20:15:02.288003634 +0000 UTC m=+1583.285692534"
Nov 02 20:15:02.395369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.395274    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:02.401727 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.401681    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb" podStartSLOduration=2.401667503 podStartE2EDuration="2.401667503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.398471746 +0000 UTC m=+1583.396160358" watchObservedRunningTime="2026-11-02 20:15:02.401667503 +0000 UTC m=+1583.399356112"
Nov 02 20:15:02.516320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.516267    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2" podStartSLOduration=2.516252895 podStartE2EDuration="2.516252895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.512736366 +0000 UTC m=+1583.510424976" watchObservedRunningTime="2026-11-02 20:15:02.516252895 +0000 UTC m=+1583.513941504"
Nov 02 20:15:02.612153 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.612116    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:02.725055 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.724954    5224 generic.go:356] "Generic (PLEG): container finished" podID="63752026-3fce-43fd-8eb5-b0dfb4e4c398" containerID="6a53a4766dbc48b22380636a47ff698dd776878acedc5cb98f0bd651d14b13f0" exitCode=60
Nov 02 20:15:02.725509 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.725053    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f" event={"ID":"63752026-3fce-43fd-8eb5-b0dfb4e4c398","Type":"ContainerDied","Data":"6a53a4766dbc48b22380636a47ff698dd776878acedc5cb98f0bd651d14b13f0"}
Nov 02 20:15:02.725509 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.725359    5224 scope.go:122] "RemoveContainer" containerID="6a53a4766dbc48b22380636a47ff698dd776878acedc5cb98f0bd651d14b13f0"
Nov 02 20:15:02.727021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.726994    5224 generic.go:356] "Generic (PLEG): container finished" podID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a" containerID="acee544c24adbf1cabaa8968d383cd0c9c257235d27b74edf0fff50c52fa508b" exitCode=60
Nov 02 20:15:02.727152 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.727100    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2" event={"ID":"dbf16df4-48e0-417b-8a03-82cf8fb4a10a","Type":"ContainerDied","Data":"acee544c24adbf1cabaa8968d383cd0c9c257235d27b74edf0fff50c52fa508b"}
Nov 02 20:15:02.727365 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.727347    5224 scope.go:122] "RemoveContainer" containerID="acee544c24adbf1cabaa8968d383cd0c9c257235d27b74edf0fff50c52fa508b"
Nov 02 20:15:02.730108 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.728986    5224 generic.go:356] "Generic (PLEG): container finished" podID="2189a356-2417-4635-b1b4-22f3920a5f21" containerID="d757edff1eb3fdbbe7973999a928c3824d0d6c5ff910a5880bdb9e19300b385a" exitCode=60
Nov 02 20:15:02.730108 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.729019    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx" event={"ID":"2189a356-2417-4635-b1b4-22f3920a5f21","Type":"ContainerDied","Data":"d757edff1eb3fdbbe7973999a928c3824d0d6c5ff910a5880bdb9e19300b385a"}
Nov 02 20:15:02.730108 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.729297    5224 scope.go:122] "RemoveContainer" containerID="d757edff1eb3fdbbe7973999a928c3824d0d6c5ff910a5880bdb9e19300b385a"
Nov 02 20:15:02.730904 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.730883    5224 generic.go:356] "Generic (PLEG): container finished" podID="ed59494c-0a34-41ab-ba57-ae829d35f753" containerID="ab3b5bae6eccef31975966327310ebc0bc60a41bcd1ab67ff7526689671e2464" exitCode=60
Nov 02 20:15:02.730983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.730967    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm" event={"ID":"ed59494c-0a34-41ab-ba57-ae829d35f753","Type":"ContainerDied","Data":"ab3b5bae6eccef31975966327310ebc0bc60a41bcd1ab67ff7526689671e2464"}
Nov 02 20:15:02.731159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.731142    5224 scope.go:122] "RemoveContainer" containerID="ab3b5bae6eccef31975966327310ebc0bc60a41bcd1ab67ff7526689671e2464"
Nov 02 20:15:02.732622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.732602    5224 generic.go:356] "Generic (PLEG): container finished" podID="56b298bc-073a-49e8-86ae-998b10102bfe" containerID="e783b0d575cbfe790cae80f7281ea12d38ba4be51234dfbbea6324dc169088b8" exitCode=60
Nov 02 20:15:02.732706 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.732678    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8" event={"ID":"56b298bc-073a-49e8-86ae-998b10102bfe","Type":"ContainerDied","Data":"e783b0d575cbfe790cae80f7281ea12d38ba4be51234dfbbea6324dc169088b8"}
Nov 02 20:15:02.732884 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.732867    5224 scope.go:122] "RemoveContainer" containerID="e783b0d575cbfe790cae80f7281ea12d38ba4be51234dfbbea6324dc169088b8"
Nov 02 20:15:02.734392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.734374    5224 generic.go:356] "Generic (PLEG): container finished" podID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" containerID="62f72ddabc48ab8e47c21145bab6e9f3ccdde544083ed276e80a976ccecbc077" exitCode=60
Nov 02 20:15:02.734520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.734457    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq" event={"ID":"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d","Type":"ContainerDied","Data":"62f72ddabc48ab8e47c21145bab6e9f3ccdde544083ed276e80a976ccecbc077"}
Nov 02 20:15:02.734670 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.734649    5224 scope.go:122] "RemoveContainer" containerID="62f72ddabc48ab8e47c21145bab6e9f3ccdde544083ed276e80a976ccecbc077"
Nov 02 20:15:02.736140 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.736120    5224 generic.go:356] "Generic (PLEG): container finished" podID="95f31166-a27b-4a71-98d0-a727faea4589" containerID="bb0d4699091fe31689230919f1ef5ce39a2db435f964bd7efa69f4cc2308c0e8" exitCode=60
Nov 02 20:15:02.736215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.736143    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb" event={"ID":"95f31166-a27b-4a71-98d0-a727faea4589","Type":"ContainerDied","Data":"bb0d4699091fe31689230919f1ef5ce39a2db435f964bd7efa69f4cc2308c0e8"}
Nov 02 20:15:02.736443 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.736422    5224 scope.go:122] "RemoveContainer" containerID="bb0d4699091fe31689230919f1ef5ce39a2db435f964bd7efa69f4cc2308c0e8"
Nov 02 20:15:02.787495 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:02.787466    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:03.750451 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:03.750397    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx" event={"ID":"2189a356-2417-4635-b1b4-22f3920a5f21","Type":"ContainerStarted","Data":"d189c7cf666459296d003ee997e21a1445f13fdfd09a5ea6618e6b9086f79642"}
Nov 02 20:15:03.753810 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:03.753457    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm" event={"ID":"ed59494c-0a34-41ab-ba57-ae829d35f753","Type":"ContainerStarted","Data":"41d25f8d25d6a5c4f79cddac6e83b078298cac83e6d3283f71f7d1bf8afccc44"}
Nov 02 20:15:03.756212 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:03.756152    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8" event={"ID":"56b298bc-073a-49e8-86ae-998b10102bfe","Type":"ContainerStarted","Data":"cff29055742ba51078ba5b12b9b3395330ac0a548a0a3088c6ef2a63b9411829"}
Nov 02 20:15:03.759006 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:03.758983    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq" event={"ID":"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d","Type":"ContainerStarted","Data":"a86a0c66d9cc1dfbf9122c6720259d2c0f7f31a27bb533135b636f126774ee44"}
Nov 02 20:15:03.761779 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:03.761639    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb" event={"ID":"95f31166-a27b-4a71-98d0-a727faea4589","Type":"ContainerStarted","Data":"5c21e27f219a8459168b62b79a2d7e1e203bfac1d00983415afa57912966fc3c"}
Nov 02 20:15:03.765271 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:03.765255    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f" event={"ID":"63752026-3fce-43fd-8eb5-b0dfb4e4c398","Type":"ContainerStarted","Data":"afde1840c90b94a267ad09ad8b887b0244120faa58ec54f7ea0d5eaec9ac0e0f"}
Nov 02 20:15:03.765551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:03.765530    5224 scope.go:122] "RemoveContainer" containerID="afde1840c90b94a267ad09ad8b887b0244120faa58ec54f7ea0d5eaec9ac0e0f"
Nov 02 20:15:03.765852 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:03.765818    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f_odh-ai-gateway-infra(63752026-3fce-43fd-8eb5-b0dfb4e4c398)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398"
Nov 02 20:15:03.767191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:03.767165    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2" event={"ID":"dbf16df4-48e0-417b-8a03-82cf8fb4a10a","Type":"ContainerStarted","Data":"ca6bbb2f5beecba4111f427b264a797db1eef034a698d70f9872b65bf62ec185"}
Nov 02 20:15:04.775855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.775824    5224 generic.go:356] "Generic (PLEG): container finished" podID="ed59494c-0a34-41ab-ba57-ae829d35f753" containerID="41d25f8d25d6a5c4f79cddac6e83b078298cac83e6d3283f71f7d1bf8afccc44" exitCode=60
Nov 02 20:15:04.776345 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.775906    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm" event={"ID":"ed59494c-0a34-41ab-ba57-ae829d35f753","Type":"ContainerDied","Data":"41d25f8d25d6a5c4f79cddac6e83b078298cac83e6d3283f71f7d1bf8afccc44"}
Nov 02 20:15:04.776345 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.775967    5224 scope.go:122] "RemoveContainer" containerID="ab3b5bae6eccef31975966327310ebc0bc60a41bcd1ab67ff7526689671e2464"
Nov 02 20:15:04.776345 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.776138    5224 scope.go:122] "RemoveContainer" containerID="41d25f8d25d6a5c4f79cddac6e83b078298cac83e6d3283f71f7d1bf8afccc44"
Nov 02 20:15:04.776469 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:04.776434    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm_odh-ai-gateway-infra(ed59494c-0a34-41ab-ba57-ae829d35f753)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm" podUID="ed59494c-0a34-41ab-ba57-ae829d35f753"
Nov 02 20:15:04.778064 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.778045    5224 generic.go:356] "Generic (PLEG): container finished" podID="56b298bc-073a-49e8-86ae-998b10102bfe" containerID="cff29055742ba51078ba5b12b9b3395330ac0a548a0a3088c6ef2a63b9411829" exitCode=60
Nov 02 20:15:04.778180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.778115    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8" event={"ID":"56b298bc-073a-49e8-86ae-998b10102bfe","Type":"ContainerDied","Data":"cff29055742ba51078ba5b12b9b3395330ac0a548a0a3088c6ef2a63b9411829"}
Nov 02 20:15:04.778292 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.778278    5224 scope.go:122] "RemoveContainer" containerID="cff29055742ba51078ba5b12b9b3395330ac0a548a0a3088c6ef2a63b9411829"
Nov 02 20:15:04.778527 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:04.778500    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8_odh-ai-gateway-infra(56b298bc-073a-49e8-86ae-998b10102bfe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8" podUID="56b298bc-073a-49e8-86ae-998b10102bfe"
Nov 02 20:15:04.779998 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.779977    5224 generic.go:356] "Generic (PLEG): container finished" podID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" containerID="a86a0c66d9cc1dfbf9122c6720259d2c0f7f31a27bb533135b636f126774ee44" exitCode=60
Nov 02 20:15:04.780093 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.780003    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq" event={"ID":"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d","Type":"ContainerDied","Data":"a86a0c66d9cc1dfbf9122c6720259d2c0f7f31a27bb533135b636f126774ee44"}
Nov 02 20:15:04.780267 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.780251    5224 scope.go:122] "RemoveContainer" containerID="a86a0c66d9cc1dfbf9122c6720259d2c0f7f31a27bb533135b636f126774ee44"
Nov 02 20:15:04.780501 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:04.780480    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq_odh-ai-gateway-infra(d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq" podUID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d"
Nov 02 20:15:04.782059 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.782038    5224 generic.go:356] "Generic (PLEG): container finished" podID="95f31166-a27b-4a71-98d0-a727faea4589" containerID="5c21e27f219a8459168b62b79a2d7e1e203bfac1d00983415afa57912966fc3c" exitCode=60
Nov 02 20:15:04.782136 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.782114    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb" event={"ID":"95f31166-a27b-4a71-98d0-a727faea4589","Type":"ContainerDied","Data":"5c21e27f219a8459168b62b79a2d7e1e203bfac1d00983415afa57912966fc3c"}
Nov 02 20:15:04.782281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.782263    5224 scope.go:122] "RemoveContainer" containerID="5c21e27f219a8459168b62b79a2d7e1e203bfac1d00983415afa57912966fc3c"
Nov 02 20:15:04.782516 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:04.782493    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb_odh-ai-gateway-infra(95f31166-a27b-4a71-98d0-a727faea4589)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb" podUID="95f31166-a27b-4a71-98d0-a727faea4589"
Nov 02 20:15:04.784277 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.784253    5224 generic.go:356] "Generic (PLEG): container finished" podID="63752026-3fce-43fd-8eb5-b0dfb4e4c398" containerID="afde1840c90b94a267ad09ad8b887b0244120faa58ec54f7ea0d5eaec9ac0e0f" exitCode=60
Nov 02 20:15:04.784370 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.784331    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f" event={"ID":"63752026-3fce-43fd-8eb5-b0dfb4e4c398","Type":"ContainerDied","Data":"afde1840c90b94a267ad09ad8b887b0244120faa58ec54f7ea0d5eaec9ac0e0f"}
Nov 02 20:15:04.784556 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.784538    5224 scope.go:122] "RemoveContainer" containerID="afde1840c90b94a267ad09ad8b887b0244120faa58ec54f7ea0d5eaec9ac0e0f"
Nov 02 20:15:04.784809 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:04.784789    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f_odh-ai-gateway-infra(63752026-3fce-43fd-8eb5-b0dfb4e4c398)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398"
Nov 02 20:15:04.786191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.786170    5224 generic.go:356] "Generic (PLEG): container finished" podID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a" containerID="ca6bbb2f5beecba4111f427b264a797db1eef034a698d70f9872b65bf62ec185" exitCode=60
Nov 02 20:15:04.786285 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.786220    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2" event={"ID":"dbf16df4-48e0-417b-8a03-82cf8fb4a10a","Type":"ContainerDied","Data":"ca6bbb2f5beecba4111f427b264a797db1eef034a698d70f9872b65bf62ec185"}
Nov 02 20:15:04.786460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.786444    5224 scope.go:122] "RemoveContainer" containerID="ca6bbb2f5beecba4111f427b264a797db1eef034a698d70f9872b65bf62ec185"
Nov 02 20:15:04.786682 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:04.786660    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894175-9zws2_odh-ai-gateway-infra(dbf16df4-48e0-417b-8a03-82cf8fb4a10a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2" podUID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a"
Nov 02 20:15:04.788295 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.788275    5224 generic.go:356] "Generic (PLEG): container finished" podID="2189a356-2417-4635-b1b4-22f3920a5f21" containerID="d189c7cf666459296d003ee997e21a1445f13fdfd09a5ea6618e6b9086f79642" exitCode=60
Nov 02 20:15:04.788377 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.788317    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx" event={"ID":"2189a356-2417-4635-b1b4-22f3920a5f21","Type":"ContainerDied","Data":"d189c7cf666459296d003ee997e21a1445f13fdfd09a5ea6618e6b9086f79642"}
Nov 02 20:15:04.788568 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.788552    5224 scope.go:122] "RemoveContainer" containerID="d189c7cf666459296d003ee997e21a1445f13fdfd09a5ea6618e6b9086f79642"
Nov 02 20:15:04.788832 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:04.788812    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx_odh-ai-gateway-infra(2189a356-2417-4635-b1b4-22f3920a5f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx" podUID="2189a356-2417-4635-b1b4-22f3920a5f21"
Nov 02 20:15:04.790342 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.790312    5224 scope.go:122] "RemoveContainer" containerID="e783b0d575cbfe790cae80f7281ea12d38ba4be51234dfbbea6324dc169088b8"
Nov 02 20:15:04.804206 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.804185    5224 scope.go:122] "RemoveContainer" containerID="62f72ddabc48ab8e47c21145bab6e9f3ccdde544083ed276e80a976ccecbc077"
Nov 02 20:15:04.817671 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.817650    5224 scope.go:122] "RemoveContainer" containerID="bb0d4699091fe31689230919f1ef5ce39a2db435f964bd7efa69f4cc2308c0e8"
Nov 02 20:15:04.831867 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.831848    5224 scope.go:122] "RemoveContainer" containerID="6a53a4766dbc48b22380636a47ff698dd776878acedc5cb98f0bd651d14b13f0"
Nov 02 20:15:04.844884 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.844863    5224 scope.go:122] "RemoveContainer" containerID="acee544c24adbf1cabaa8968d383cd0c9c257235d27b74edf0fff50c52fa508b"
Nov 02 20:15:04.857854 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:04.857835    5224 scope.go:122] "RemoveContainer" containerID="d757edff1eb3fdbbe7973999a928c3824d0d6c5ff910a5880bdb9e19300b385a"
Nov 02 20:15:05.798358 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:05.798327    5224 scope.go:122] "RemoveContainer" containerID="afde1840c90b94a267ad09ad8b887b0244120faa58ec54f7ea0d5eaec9ac0e0f"
Nov 02 20:15:05.798814 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:05.798684    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f_odh-ai-gateway-infra(63752026-3fce-43fd-8eb5-b0dfb4e4c398)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398"
Nov 02 20:15:05.799805 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:05.799783    5224 scope.go:122] "RemoveContainer" containerID="ca6bbb2f5beecba4111f427b264a797db1eef034a698d70f9872b65bf62ec185"
Nov 02 20:15:05.800062 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:05.800042    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894175-9zws2_odh-ai-gateway-infra(dbf16df4-48e0-417b-8a03-82cf8fb4a10a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2" podUID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a"
Nov 02 20:15:05.801260 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:05.801242    5224 scope.go:122] "RemoveContainer" containerID="d189c7cf666459296d003ee997e21a1445f13fdfd09a5ea6618e6b9086f79642"
Nov 02 20:15:05.801485 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:05.801457    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx_odh-ai-gateway-infra(2189a356-2417-4635-b1b4-22f3920a5f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx" podUID="2189a356-2417-4635-b1b4-22f3920a5f21"
Nov 02 20:15:05.802734 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:05.802717    5224 scope.go:122] "RemoveContainer" containerID="41d25f8d25d6a5c4f79cddac6e83b078298cac83e6d3283f71f7d1bf8afccc44"
Nov 02 20:15:05.802978 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:05.802957    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm_odh-ai-gateway-infra(ed59494c-0a34-41ab-ba57-ae829d35f753)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm" podUID="ed59494c-0a34-41ab-ba57-ae829d35f753"
Nov 02 20:15:05.804311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:05.804291    5224 scope.go:122] "RemoveContainer" containerID="cff29055742ba51078ba5b12b9b3395330ac0a548a0a3088c6ef2a63b9411829"
Nov 02 20:15:05.804622 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:05.804502    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8_odh-ai-gateway-infra(56b298bc-073a-49e8-86ae-998b10102bfe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8" podUID="56b298bc-073a-49e8-86ae-998b10102bfe"
Nov 02 20:15:05.805975 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:05.805958    5224 scope.go:122] "RemoveContainer" containerID="a86a0c66d9cc1dfbf9122c6720259d2c0f7f31a27bb533135b636f126774ee44"
Nov 02 20:15:05.806183 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:05.806161    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq_odh-ai-gateway-infra(d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq" podUID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d"
Nov 02 20:15:05.807362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:05.807345    5224 scope.go:122] "RemoveContainer" containerID="5c21e27f219a8459168b62b79a2d7e1e203bfac1d00983415afa57912966fc3c"
Nov 02 20:15:05.807563 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:05.807543    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb_odh-ai-gateway-infra(95f31166-a27b-4a71-98d0-a727faea4589)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb" podUID="95f31166-a27b-4a71-98d0-a727faea4589"
Nov 02 20:15:07.289379 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:07.289053    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:09.074907 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:09.074846    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:12.180059 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:12.180022    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"]
Nov 02 20:15:12.180471 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:12.180222    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" podUID="21980784-069f-41e3-ae48-40cd1c32186c" containerName="istio-proxy" containerID="cri-o://dec755234b67b5dac7818696adc61641efc8bf76a54ec3f03a54fb6ba82a0305" gracePeriod=30
Nov 02 20:15:14.169795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:14.168351    5224 scope.go:122] "RemoveContainer" containerID="cff29055742ba51078ba5b12b9b3395330ac0a548a0a3088c6ef2a63b9411829"
Nov 02 20:15:14.282536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:14.282503    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:14.630373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:14.630339    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:14.820370 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:14.820329    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:14.976465 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:14.976430    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8" event={"ID":"56b298bc-073a-49e8-86ae-998b10102bfe","Type":"ContainerStarted","Data":"4cbb0f71c26ffb10be2acbb8314fb6be833e75b952bbe124fc2cefa0238be349"}
Nov 02 20:15:15.174946 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:15.172382    5224 scope.go:122] "RemoveContainer" containerID="a86a0c66d9cc1dfbf9122c6720259d2c0f7f31a27bb533135b636f126774ee44"
Nov 02 20:15:15.174946 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:15.174436    5224 scope.go:122] "RemoveContainer" containerID="5c21e27f219a8459168b62b79a2d7e1e203bfac1d00983415afa57912966fc3c"
Nov 02 20:15:15.175832 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:15.175027    5224 scope.go:122] "RemoveContainer" containerID="afde1840c90b94a267ad09ad8b887b0244120faa58ec54f7ea0d5eaec9ac0e0f"
Nov 02 20:15:15.987000 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:15.986943    5224 generic.go:356] "Generic (PLEG): container finished" podID="56b298bc-073a-49e8-86ae-998b10102bfe" containerID="4cbb0f71c26ffb10be2acbb8314fb6be833e75b952bbe124fc2cefa0238be349" exitCode=60
Nov 02 20:15:15.987209 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:15.987042    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8" event={"ID":"56b298bc-073a-49e8-86ae-998b10102bfe","Type":"ContainerDied","Data":"4cbb0f71c26ffb10be2acbb8314fb6be833e75b952bbe124fc2cefa0238be349"}
Nov 02 20:15:15.987209 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:15.987099    5224 scope.go:122] "RemoveContainer" containerID="cff29055742ba51078ba5b12b9b3395330ac0a548a0a3088c6ef2a63b9411829"
Nov 02 20:15:15.987618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:15.987587    5224 scope.go:122] "RemoveContainer" containerID="4cbb0f71c26ffb10be2acbb8314fb6be833e75b952bbe124fc2cefa0238be349"
Nov 02 20:15:15.988916 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:15.988890    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8_odh-ai-gateway-infra(56b298bc-073a-49e8-86ae-998b10102bfe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8" podUID="56b298bc-073a-49e8-86ae-998b10102bfe"
Nov 02 20:15:15.992829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:15.990925    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq" event={"ID":"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d","Type":"ContainerStarted","Data":"5740293c75643c608f98e1cdb43e66204b809f4cc564c59421fefeff9f31c445"}
Nov 02 20:15:15.996993 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:15.996960    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb" event={"ID":"95f31166-a27b-4a71-98d0-a727faea4589","Type":"ContainerStarted","Data":"c5164b9a43c8821f40beca234c63990b6bb9cb0a8e8fe35500cc86e8bdb3e4a7"}
Nov 02 20:15:15.999038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:15.999017    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f" event={"ID":"63752026-3fce-43fd-8eb5-b0dfb4e4c398","Type":"ContainerStarted","Data":"cf88af0eeb70f6c2538ae26892968e0070e725b6daf727dd712ed908319d33b5"}
Nov 02 20:15:16.109469 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:16.109436    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"]
Nov 02 20:15:16.397330 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:16.397295    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:16.844847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:16.844528    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"]
Nov 02 20:15:16.844847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:16.844829    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" containerName="istio-proxy" containerID="cri-o://28da88f973b102a70d58af30b5581b14df16bd78b11b377d4733a9979bf82600" gracePeriod=30
Nov 02 20:15:16.980792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:16.980440    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:17.072427 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.072383    5224 generic.go:356] "Generic (PLEG): container finished" podID="63752026-3fce-43fd-8eb5-b0dfb4e4c398" containerID="cf88af0eeb70f6c2538ae26892968e0070e725b6daf727dd712ed908319d33b5" exitCode=60
Nov 02 20:15:17.072987 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.072948    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f" event={"ID":"63752026-3fce-43fd-8eb5-b0dfb4e4c398","Type":"ContainerDied","Data":"cf88af0eeb70f6c2538ae26892968e0070e725b6daf727dd712ed908319d33b5"}
Nov 02 20:15:17.073280 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.073261    5224 scope.go:122] "RemoveContainer" containerID="afde1840c90b94a267ad09ad8b887b0244120faa58ec54f7ea0d5eaec9ac0e0f"
Nov 02 20:15:17.073934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.073915    5224 scope.go:122] "RemoveContainer" containerID="cf88af0eeb70f6c2538ae26892968e0070e725b6daf727dd712ed908319d33b5"
Nov 02 20:15:17.087795 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:17.074415    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f_odh-ai-gateway-infra(63752026-3fce-43fd-8eb5-b0dfb4e4c398)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398"
Nov 02 20:15:17.095846 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.092017    5224 generic.go:356] "Generic (PLEG): container finished" podID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" containerID="5740293c75643c608f98e1cdb43e66204b809f4cc564c59421fefeff9f31c445" exitCode=60
Nov 02 20:15:17.095846 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.092133    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq" event={"ID":"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d","Type":"ContainerDied","Data":"5740293c75643c608f98e1cdb43e66204b809f4cc564c59421fefeff9f31c445"}
Nov 02 20:15:17.095846 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.093019    5224 scope.go:122] "RemoveContainer" containerID="5740293c75643c608f98e1cdb43e66204b809f4cc564c59421fefeff9f31c445"
Nov 02 20:15:17.110793 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:17.109522    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq_odh-ai-gateway-infra(d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq" podUID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d"
Nov 02 20:15:17.117331 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.117286    5224 generic.go:356] "Generic (PLEG): container finished" podID="95f31166-a27b-4a71-98d0-a727faea4589" containerID="c5164b9a43c8821f40beca234c63990b6bb9cb0a8e8fe35500cc86e8bdb3e4a7" exitCode=60
Nov 02 20:15:17.119796 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.119668    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb" event={"ID":"95f31166-a27b-4a71-98d0-a727faea4589","Type":"ContainerDied","Data":"c5164b9a43c8821f40beca234c63990b6bb9cb0a8e8fe35500cc86e8bdb3e4a7"}
Nov 02 20:15:17.120871 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.120756    5224 scope.go:122] "RemoveContainer" containerID="c5164b9a43c8821f40beca234c63990b6bb9cb0a8e8fe35500cc86e8bdb3e4a7"
Nov 02 20:15:17.122138 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:17.122100    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb_odh-ai-gateway-infra(95f31166-a27b-4a71-98d0-a727faea4589)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb" podUID="95f31166-a27b-4a71-98d0-a727faea4589"
Nov 02 20:15:17.170049 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.167910    5224 scope.go:122] "RemoveContainer" containerID="41d25f8d25d6a5c4f79cddac6e83b078298cac83e6d3283f71f7d1bf8afccc44"
Nov 02 20:15:17.170282 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.170255    5224 scope.go:122] "RemoveContainer" containerID="a86a0c66d9cc1dfbf9122c6720259d2c0f7f31a27bb533135b636f126774ee44"
Nov 02 20:15:17.171246 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.170911    5224 scope.go:122] "RemoveContainer" containerID="ca6bbb2f5beecba4111f427b264a797db1eef034a698d70f9872b65bf62ec185"
Nov 02 20:15:17.171779 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.171740    5224 scope.go:122] "RemoveContainer" containerID="d189c7cf666459296d003ee997e21a1445f13fdfd09a5ea6618e6b9086f79642"
Nov 02 20:15:17.238814 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.238713    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"]
Nov 02 20:15:17.334477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.334455    5224 scope.go:122] "RemoveContainer" containerID="5c21e27f219a8459168b62b79a2d7e1e203bfac1d00983415afa57912966fc3c"
Nov 02 20:15:17.359440 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.359127    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"]
Nov 02 20:15:17.473203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.472517    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"]
Nov 02 20:15:17.480853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.480823    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"
Nov 02 20:15:17.483980 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.483955    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:15:17.528965 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.528926    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-certs\") pod \"21980784-069f-41e3-ae48-40cd1c32186c\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") "
Nov 02 20:15:17.529160 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529018    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56b298bc-073a-49e8-86ae-998b10102bfe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b298bc-073a-49e8-86ae-998b10102bfe-maas-api-service-ca\") pod \"56b298bc-073a-49e8-86ae-998b10102bfe\" (UID: \"56b298bc-073a-49e8-86ae-998b10102bfe\") "
Nov 02 20:15:17.529160 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529071    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b298bc-073a-49e8-86ae-998b10102bfe-kube-api-access-4k2wp\" (UniqueName: \"kubernetes.io/projected/56b298bc-073a-49e8-86ae-998b10102bfe-kube-api-access-4k2wp\") pod \"56b298bc-073a-49e8-86ae-998b10102bfe\" (UID: \"56b298bc-073a-49e8-86ae-998b10102bfe\") "
Nov 02 20:15:17.529160 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529148    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21980784-069f-41e3-ae48-40cd1c32186c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21980784-069f-41e3-ae48-40cd1c32186c-istiod-ca-cert\") pod \"21980784-069f-41e3-ae48-40cd1c32186c\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") "
Nov 02 20:15:17.529339 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529191    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-data\") pod \"21980784-069f-41e3-ae48-40cd1c32186c\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") "
Nov 02 20:15:17.529339 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529221    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-socket\") pod \"21980784-069f-41e3-ae48-40cd1c32186c\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") "
Nov 02 20:15:17.529339 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529269    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-kube-api-access-4rxgs\" (UniqueName: \"kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-kube-api-access-4rxgs\") pod \"21980784-069f-41e3-ae48-40cd1c32186c\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") "
Nov 02 20:15:17.529339 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529258    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-certs" pod "21980784-069f-41e3-ae48-40cd1c32186c" (UID: "21980784-069f-41e3-ae48-40cd1c32186c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:17.529536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529361    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-envoy\") pod \"21980784-069f-41e3-ae48-40cd1c32186c\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") "
Nov 02 20:15:17.529536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529431    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-istio-token\" (UniqueName: \"kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-istio-token\") pod \"21980784-069f-41e3-ae48-40cd1c32186c\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") "
Nov 02 20:15:17.529536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529496    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b298bc-073a-49e8-86ae-998b10102bfe-maas-api-service-ca" pod "56b298bc-073a-49e8-86ae-998b10102bfe" (UID: "56b298bc-073a-49e8-86ae-998b10102bfe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:17.529536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529505    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-socket" pod "21980784-069f-41e3-ae48-40cd1c32186c" (UID: "21980784-069f-41e3-ae48-40cd1c32186c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:17.529536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529518    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-credential-socket\") pod \"21980784-069f-41e3-ae48-40cd1c32186c\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") "
Nov 02 20:15:17.529782 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529561    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21980784-069f-41e3-ae48-40cd1c32186c-istiod-ca-cert" pod "21980784-069f-41e3-ae48-40cd1c32186c" (UID: "21980784-069f-41e3-ae48-40cd1c32186c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:17.529782 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529584    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21980784-069f-41e3-ae48-40cd1c32186c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21980784-069f-41e3-ae48-40cd1c32186c-istio-podinfo\") pod \"21980784-069f-41e3-ae48-40cd1c32186c\" (UID: \"21980784-069f-41e3-ae48-40cd1c32186c\") "
Nov 02 20:15:17.529889 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529790    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-data" pod "21980784-069f-41e3-ae48-40cd1c32186c" (UID: "21980784-069f-41e3-ae48-40cd1c32186c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:17.529889 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.529860    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-credential-socket" pod "21980784-069f-41e3-ae48-40cd1c32186c" (UID: "21980784-069f-41e3-ae48-40cd1c32186c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:17.530312 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.530281    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21980784-069f-41e3-ae48-40cd1c32186c-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:17.530434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.530316    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:17.530434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.530332    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:17.530434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.530346    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:17.530434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.530360    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:17.530434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.530375    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b298bc-073a-49e8-86ae-998b10102bfe-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:17.553380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.553341    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-envoy" pod "21980784-069f-41e3-ae48-40cd1c32186c" (UID: "21980784-069f-41e3-ae48-40cd1c32186c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:17.553380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.553365    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b298bc-073a-49e8-86ae-998b10102bfe-kube-api-access-4k2wp" pod "56b298bc-073a-49e8-86ae-998b10102bfe" (UID: "56b298bc-073a-49e8-86ae-998b10102bfe"). InnerVolumeSpecName "kube-api-access-4k2wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:17.553380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.553341    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-istio-token" pod "21980784-069f-41e3-ae48-40cd1c32186c" (UID: "21980784-069f-41e3-ae48-40cd1c32186c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:17.553680 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.553428    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-kube-api-access-4rxgs" pod "21980784-069f-41e3-ae48-40cd1c32186c" (UID: "21980784-069f-41e3-ae48-40cd1c32186c"). InnerVolumeSpecName "kube-api-access-4rxgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:17.553680 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.553608    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21980784-069f-41e3-ae48-40cd1c32186c-istio-podinfo" pod "21980784-069f-41e3-ae48-40cd1c32186c" (UID: "21980784-069f-41e3-ae48-40cd1c32186c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:15:17.632334 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.632254    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21980784-069f-41e3-ae48-40cd1c32186c-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:17.632334 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.632283    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k2wp\" (UniqueName: \"kubernetes.io/projected/56b298bc-073a-49e8-86ae-998b10102bfe-kube-api-access-4k2wp\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:17.632334 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.632293    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rxgs\" (UniqueName: \"kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-kube-api-access-4rxgs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:17.632334 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.632302    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21980784-069f-41e3-ae48-40cd1c32186c-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:17.632334 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.632311    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21980784-069f-41e3-ae48-40cd1c32186c-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:17.730372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:17.730335    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:18.129202 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.129153    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2" event={"ID":"dbf16df4-48e0-417b-8a03-82cf8fb4a10a","Type":"ContainerStarted","Data":"242d5f0582a33e336839df1a76ea445fb8adef906bdf561f797218b00cb2c2d6"}
Nov 02 20:15:18.131430 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.131399    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx" event={"ID":"2189a356-2417-4635-b1b4-22f3920a5f21","Type":"ContainerStarted","Data":"60d4606e9d0c0f103889f12f022228744b7cd422fb440f3f11df03f998b6b4c2"}
Nov 02 20:15:18.133477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.133450    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm" event={"ID":"ed59494c-0a34-41ab-ba57-ae829d35f753","Type":"ContainerStarted","Data":"854fad10faf99e5761b29ed32d7682155b38867a4b5f97a61be161d7f9a921e3"}
Nov 02 20:15:18.135027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.135009    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"
Nov 02 20:15:18.135027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.135017    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8" event={"ID":"56b298bc-073a-49e8-86ae-998b10102bfe","Type":"ContainerDied","Data":"758369e3b9331f9f551d3e204abb094d2ced0bc61b1177edca1c69ea8abe2701"}
Nov 02 20:15:18.135177 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.135073    5224 scope.go:122] "RemoveContainer" containerID="4cbb0f71c26ffb10be2acbb8314fb6be833e75b952bbe124fc2cefa0238be349"
Nov 02 20:15:18.144736 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:18.144710    5224 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b298bc_073a_49e8_86ae_998b10102bfe.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 20:15:18.158328 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.158298    5224 generic.go:356] "Generic (PLEG): container finished" podID="21980784-069f-41e3-ae48-40cd1c32186c" containerID="dec755234b67b5dac7818696adc61641efc8bf76a54ec3f03a54fb6ba82a0305" exitCode=0
Nov 02 20:15:18.158480 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.158349    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" event={"ID":"21980784-069f-41e3-ae48-40cd1c32186c","Type":"ContainerDied","Data":"dec755234b67b5dac7818696adc61641efc8bf76a54ec3f03a54fb6ba82a0305"}
Nov 02 20:15:18.158480 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.158387    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4" event={"ID":"21980784-069f-41e3-ae48-40cd1c32186c","Type":"ContainerDied","Data":"12728d4081c6cc6f5ed28ef724792b010ae73e735de0df7924549fac6030821d"}
Nov 02 20:15:18.158480 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.158448    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"
Nov 02 20:15:18.165547 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.165427    5224 scope.go:122] "RemoveContainer" containerID="dec755234b67b5dac7818696adc61641efc8bf76a54ec3f03a54fb6ba82a0305"
Nov 02 20:15:18.177956 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.177927    5224 scope.go:122] "RemoveContainer" containerID="dec755234b67b5dac7818696adc61641efc8bf76a54ec3f03a54fb6ba82a0305"
Nov 02 20:15:18.178353 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:18.178326    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec755234b67b5dac7818696adc61641efc8bf76a54ec3f03a54fb6ba82a0305\": container with ID starting with dec755234b67b5dac7818696adc61641efc8bf76a54ec3f03a54fb6ba82a0305 not found: ID does not exist" containerID="dec755234b67b5dac7818696adc61641efc8bf76a54ec3f03a54fb6ba82a0305"
Nov 02 20:15:18.178500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.178365    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec755234b67b5dac7818696adc61641efc8bf76a54ec3f03a54fb6ba82a0305"} err="failed to get container status \"dec755234b67b5dac7818696adc61641efc8bf76a54ec3f03a54fb6ba82a0305\": rpc error: code = NotFound desc = could not find container \"dec755234b67b5dac7818696adc61641efc8bf76a54ec3f03a54fb6ba82a0305\": container with ID starting with dec755234b67b5dac7818696adc61641efc8bf76a54ec3f03a54fb6ba82a0305 not found: ID does not exist"
Nov 02 20:15:18.314117 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.314085    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"
Nov 02 20:15:18.442004 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.441974    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"
Nov 02 20:15:18.446099 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.446072    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"
Nov 02 20:15:18.448597 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.448573    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63752026-3fce-43fd-8eb5-b0dfb4e4c398-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63752026-3fce-43fd-8eb5-b0dfb4e4c398-maas-api-service-ca\") pod \"63752026-3fce-43fd-8eb5-b0dfb4e4c398\" (UID: \"63752026-3fce-43fd-8eb5-b0dfb4e4c398\") "
Nov 02 20:15:18.448665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.448622    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63752026-3fce-43fd-8eb5-b0dfb4e4c398-kube-api-access-fkkjs\" (UniqueName: \"kubernetes.io/projected/63752026-3fce-43fd-8eb5-b0dfb4e4c398-kube-api-access-fkkjs\") pod \"63752026-3fce-43fd-8eb5-b0dfb4e4c398\" (UID: \"63752026-3fce-43fd-8eb5-b0dfb4e4c398\") "
Nov 02 20:15:18.449068 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.449041    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63752026-3fce-43fd-8eb5-b0dfb4e4c398-maas-api-service-ca" pod "63752026-3fce-43fd-8eb5-b0dfb4e4c398" (UID: "63752026-3fce-43fd-8eb5-b0dfb4e4c398"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:18.449203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.449174    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63752026-3fce-43fd-8eb5-b0dfb4e4c398-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:18.451162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.450919    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63752026-3fce-43fd-8eb5-b0dfb4e4c398-kube-api-access-fkkjs" pod "63752026-3fce-43fd-8eb5-b0dfb4e4c398" (UID: "63752026-3fce-43fd-8eb5-b0dfb4e4c398"). InnerVolumeSpecName "kube-api-access-fkkjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:18.490433 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.490396    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"]
Nov 02 20:15:18.493554 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.493527    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-9b8v4"]
Nov 02 20:15:18.550363 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.550271    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f31166-a27b-4a71-98d0-a727faea4589-kube-api-access-rqxbd\" (UniqueName: \"kubernetes.io/projected/95f31166-a27b-4a71-98d0-a727faea4589-kube-api-access-rqxbd\") pod \"95f31166-a27b-4a71-98d0-a727faea4589\" (UID: \"95f31166-a27b-4a71-98d0-a727faea4589\") "
Nov 02 20:15:18.550527 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.550371    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-kube-api-access-vmk98\" (UniqueName: \"kubernetes.io/projected/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-kube-api-access-vmk98\") pod \"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d\" (UID: \"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d\") "
Nov 02 20:15:18.550527 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.550406    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-maas-api-service-ca\") pod \"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d\" (UID: \"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d\") "
Nov 02 20:15:18.550658 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.550570    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f31166-a27b-4a71-98d0-a727faea4589-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f31166-a27b-4a71-98d0-a727faea4589-maas-api-service-ca\") pod \"95f31166-a27b-4a71-98d0-a727faea4589\" (UID: \"95f31166-a27b-4a71-98d0-a727faea4589\") "
Nov 02 20:15:18.550882 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.550853    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-maas-api-service-ca" pod "d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" (UID: "d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:18.551007 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.550942    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f31166-a27b-4a71-98d0-a727faea4589-maas-api-service-ca" pod "95f31166-a27b-4a71-98d0-a727faea4589" (UID: "95f31166-a27b-4a71-98d0-a727faea4589"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:18.551215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.551198    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:18.551280 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.551221    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkkjs\" (UniqueName: \"kubernetes.io/projected/63752026-3fce-43fd-8eb5-b0dfb4e4c398-kube-api-access-fkkjs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:18.551280 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.551236    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f31166-a27b-4a71-98d0-a727faea4589-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:18.552593 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.552568    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f31166-a27b-4a71-98d0-a727faea4589-kube-api-access-rqxbd" pod "95f31166-a27b-4a71-98d0-a727faea4589" (UID: "95f31166-a27b-4a71-98d0-a727faea4589"). InnerVolumeSpecName "kube-api-access-rqxbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:18.552699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.552592    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-kube-api-access-vmk98" pod "d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" (UID: "d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d"). InnerVolumeSpecName "kube-api-access-vmk98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:18.603890 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.603856    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"]
Nov 02 20:15:18.607035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.607007    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894175-k2ln8"]
Nov 02 20:15:18.652829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.652752    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqxbd\" (UniqueName: \"kubernetes.io/projected/95f31166-a27b-4a71-98d0-a727faea4589-kube-api-access-rqxbd\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:18.652829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:18.652826    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmk98\" (UniqueName: \"kubernetes.io/projected/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d-kube-api-access-vmk98\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:19.048204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.048165    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:19.180790 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.176587    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"
Nov 02 20:15:19.198810 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.196266    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21980784-069f-41e3-ae48-40cd1c32186c" path="/var/lib/kubelet/pods/21980784-069f-41e3-ae48-40cd1c32186c/volumes"
Nov 02 20:15:19.198810 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.196896    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b298bc-073a-49e8-86ae-998b10102bfe" path="/var/lib/kubelet/pods/56b298bc-073a-49e8-86ae-998b10102bfe/volumes"
Nov 02 20:15:19.198810 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.197781    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq" event={"ID":"d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d","Type":"ContainerDied","Data":"29d738a810c1ad11bea78b50a32a30a3abc58fa9bcde6beadd0c3ee09914614a"}
Nov 02 20:15:19.198810 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.197847    5224 scope.go:122] "RemoveContainer" containerID="5740293c75643c608f98e1cdb43e66204b809f4cc564c59421fefeff9f31c445"
Nov 02 20:15:19.220125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.215341    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb" event={"ID":"95f31166-a27b-4a71-98d0-a727faea4589","Type":"ContainerDied","Data":"acee708192dc99a6863ecec9dbe97c485090d2b6b71dfb072cfe7a6304dbac6e"}
Nov 02 20:15:19.220125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.215517    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"
Nov 02 20:15:19.228602 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.226036    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f" event={"ID":"63752026-3fce-43fd-8eb5-b0dfb4e4c398","Type":"ContainerDied","Data":"b3e29e75a6496f510517c535612afa3c3688420b713ad90ac5f78f4185a0d6f9"}
Nov 02 20:15:19.228602 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.226174    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"
Nov 02 20:15:19.235950 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.235545    5224 generic.go:356] "Generic (PLEG): container finished" podID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a" containerID="242d5f0582a33e336839df1a76ea445fb8adef906bdf561f797218b00cb2c2d6" exitCode=60
Nov 02 20:15:19.235950 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.235674    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2" event={"ID":"dbf16df4-48e0-417b-8a03-82cf8fb4a10a","Type":"ContainerDied","Data":"242d5f0582a33e336839df1a76ea445fb8adef906bdf561f797218b00cb2c2d6"}
Nov 02 20:15:19.241729 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.239583    5224 generic.go:356] "Generic (PLEG): container finished" podID="2189a356-2417-4635-b1b4-22f3920a5f21" containerID="60d4606e9d0c0f103889f12f022228744b7cd422fb440f3f11df03f998b6b4c2" exitCode=60
Nov 02 20:15:19.241729 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.239732    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx" event={"ID":"2189a356-2417-4635-b1b4-22f3920a5f21","Type":"ContainerDied","Data":"60d4606e9d0c0f103889f12f022228744b7cd422fb440f3f11df03f998b6b4c2"}
Nov 02 20:15:19.241729 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.240199    5224 scope.go:122] "RemoveContainer" containerID="60d4606e9d0c0f103889f12f022228744b7cd422fb440f3f11df03f998b6b4c2"
Nov 02 20:15:19.241729 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:19.240583    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx_odh-ai-gateway-infra(2189a356-2417-4635-b1b4-22f3920a5f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx" podUID="2189a356-2417-4635-b1b4-22f3920a5f21"
Nov 02 20:15:19.244494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.244274    5224 scope.go:122] "RemoveContainer" containerID="242d5f0582a33e336839df1a76ea445fb8adef906bdf561f797218b00cb2c2d6"
Nov 02 20:15:19.245060 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:19.244757    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894175-9zws2_odh-ai-gateway-infra(dbf16df4-48e0-417b-8a03-82cf8fb4a10a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2" podUID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a"
Nov 02 20:15:19.253443 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.253409    5224 generic.go:356] "Generic (PLEG): container finished" podID="ed59494c-0a34-41ab-ba57-ae829d35f753" containerID="854fad10faf99e5761b29ed32d7682155b38867a4b5f97a61be161d7f9a921e3" exitCode=60
Nov 02 20:15:19.253703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.253469    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm" event={"ID":"ed59494c-0a34-41ab-ba57-ae829d35f753","Type":"ContainerDied","Data":"854fad10faf99e5761b29ed32d7682155b38867a4b5f97a61be161d7f9a921e3"}
Nov 02 20:15:19.254380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.254360    5224 scope.go:122] "RemoveContainer" containerID="854fad10faf99e5761b29ed32d7682155b38867a4b5f97a61be161d7f9a921e3"
Nov 02 20:15:19.254961 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:19.254939    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm_odh-ai-gateway-infra(ed59494c-0a34-41ab-ba57-ae829d35f753)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm" podUID="ed59494c-0a34-41ab-ba57-ae829d35f753"
Nov 02 20:15:19.259450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.259427    5224 scope.go:122] "RemoveContainer" containerID="c5164b9a43c8821f40beca234c63990b6bb9cb0a8e8fe35500cc86e8bdb3e4a7"
Nov 02 20:15:19.287718 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.287315    5224 scope.go:122] "RemoveContainer" containerID="cf88af0eeb70f6c2538ae26892968e0070e725b6daf727dd712ed908319d33b5"
Nov 02 20:15:19.348192 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.348154    5224 scope.go:122] "RemoveContainer" containerID="ca6bbb2f5beecba4111f427b264a797db1eef034a698d70f9872b65bf62ec185"
Nov 02 20:15:19.448759 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.441169    5224 scope.go:122] "RemoveContainer" containerID="d189c7cf666459296d003ee997e21a1445f13fdfd09a5ea6618e6b9086f79642"
Nov 02 20:15:19.448759 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.443298    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"]
Nov 02 20:15:19.468570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.468547    5224 scope.go:122] "RemoveContainer" containerID="41d25f8d25d6a5c4f79cddac6e83b078298cac83e6d3283f71f7d1bf8afccc44"
Nov 02 20:15:19.728292 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.728208    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"]
Nov 02 20:15:19.842687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.842656    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"]
Nov 02 20:15:19.907250 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.907212    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"]
Nov 02 20:15:19.910281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:19.910257    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-341fb2-29894175-7h4qq"]
Nov 02 20:15:20.050779 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.050725    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"]
Nov 02 20:15:20.069829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.068439    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894175-jqklb"]
Nov 02 20:15:20.147080 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.147042    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"]
Nov 02 20:15:20.150353 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.150322    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5e04421b-29894175-v5b2f"]
Nov 02 20:15:20.449577 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.449548    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"
Nov 02 20:15:20.519342 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.519314    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"
Nov 02 20:15:20.522837 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.522813    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"
Nov 02 20:15:20.588789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.588692    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2189a356-2417-4635-b1b4-22f3920a5f21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2189a356-2417-4635-b1b4-22f3920a5f21-maas-api-service-ca\") pod \"2189a356-2417-4635-b1b4-22f3920a5f21\" (UID: \"2189a356-2417-4635-b1b4-22f3920a5f21\") "
Nov 02 20:15:20.588789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.588743    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-kube-api-access-qq6r5\" (UniqueName: \"kubernetes.io/projected/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-kube-api-access-qq6r5\") pod \"dbf16df4-48e0-417b-8a03-82cf8fb4a10a\" (UID: \"dbf16df4-48e0-417b-8a03-82cf8fb4a10a\") "
Nov 02 20:15:20.589090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.588891    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2189a356-2417-4635-b1b4-22f3920a5f21-kube-api-access-8slff\" (UniqueName: \"kubernetes.io/projected/2189a356-2417-4635-b1b4-22f3920a5f21-kube-api-access-8slff\") pod \"2189a356-2417-4635-b1b4-22f3920a5f21\" (UID: \"2189a356-2417-4635-b1b4-22f3920a5f21\") "
Nov 02 20:15:20.589090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.588976    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed59494c-0a34-41ab-ba57-ae829d35f753-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed59494c-0a34-41ab-ba57-ae829d35f753-maas-api-service-ca\") pod \"ed59494c-0a34-41ab-ba57-ae829d35f753\" (UID: \"ed59494c-0a34-41ab-ba57-ae829d35f753\") "
Nov 02 20:15:20.589090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.589070    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-maas-api-service-ca\") pod \"dbf16df4-48e0-417b-8a03-82cf8fb4a10a\" (UID: \"dbf16df4-48e0-417b-8a03-82cf8fb4a10a\") "
Nov 02 20:15:20.589238 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.589110    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed59494c-0a34-41ab-ba57-ae829d35f753-kube-api-access-q8s75\" (UniqueName: \"kubernetes.io/projected/ed59494c-0a34-41ab-ba57-ae829d35f753-kube-api-access-q8s75\") pod \"ed59494c-0a34-41ab-ba57-ae829d35f753\" (UID: \"ed59494c-0a34-41ab-ba57-ae829d35f753\") "
Nov 02 20:15:20.589238 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.589187    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2189a356-2417-4635-b1b4-22f3920a5f21-maas-api-service-ca" pod "2189a356-2417-4635-b1b4-22f3920a5f21" (UID: "2189a356-2417-4635-b1b4-22f3920a5f21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:20.589370 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.589335    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed59494c-0a34-41ab-ba57-ae829d35f753-maas-api-service-ca" pod "ed59494c-0a34-41ab-ba57-ae829d35f753" (UID: "ed59494c-0a34-41ab-ba57-ae829d35f753"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:20.589483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.589443    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-maas-api-service-ca" pod "dbf16df4-48e0-417b-8a03-82cf8fb4a10a" (UID: "dbf16df4-48e0-417b-8a03-82cf8fb4a10a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:20.589855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.589830    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2189a356-2417-4635-b1b4-22f3920a5f21-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:20.589970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.589860    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed59494c-0a34-41ab-ba57-ae829d35f753-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:20.589970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.589877    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:20.591232 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.591207    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2189a356-2417-4635-b1b4-22f3920a5f21-kube-api-access-8slff" pod "2189a356-2417-4635-b1b4-22f3920a5f21" (UID: "2189a356-2417-4635-b1b4-22f3920a5f21"). InnerVolumeSpecName "kube-api-access-8slff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:20.591323 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.591246    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-kube-api-access-qq6r5" pod "dbf16df4-48e0-417b-8a03-82cf8fb4a10a" (UID: "dbf16df4-48e0-417b-8a03-82cf8fb4a10a"). InnerVolumeSpecName "kube-api-access-qq6r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:20.591323 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.591251    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed59494c-0a34-41ab-ba57-ae829d35f753-kube-api-access-q8s75" pod "ed59494c-0a34-41ab-ba57-ae829d35f753" (UID: "ed59494c-0a34-41ab-ba57-ae829d35f753"). InnerVolumeSpecName "kube-api-access-q8s75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:20.691429 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.691389    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8slff\" (UniqueName: \"kubernetes.io/projected/2189a356-2417-4635-b1b4-22f3920a5f21-kube-api-access-8slff\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:20.691429 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.691425    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8s75\" (UniqueName: \"kubernetes.io/projected/ed59494c-0a34-41ab-ba57-ae829d35f753-kube-api-access-q8s75\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:20.691647 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:20.691440    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq6r5\" (UniqueName: \"kubernetes.io/projected/dbf16df4-48e0-417b-8a03-82cf8fb4a10a-kube-api-access-qq6r5\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:21.076528 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.076479    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:21.171809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.171727    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398" path="/var/lib/kubelet/pods/63752026-3fce-43fd-8eb5-b0dfb4e4c398/volumes"
Nov 02 20:15:21.172415 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.172394    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f31166-a27b-4a71-98d0-a727faea4589" path="/var/lib/kubelet/pods/95f31166-a27b-4a71-98d0-a727faea4589/volumes"
Nov 02 20:15:21.172836 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.172815    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" path="/var/lib/kubelet/pods/d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d/volumes"
Nov 02 20:15:21.288531 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.288499    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"
Nov 02 20:15:21.288712 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.288501    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2" event={"ID":"dbf16df4-48e0-417b-8a03-82cf8fb4a10a","Type":"ContainerDied","Data":"ed3daeddf5f63a0b0765cba82f53f2c208018faca5b6c1e14019f3b849e1d4f7"}
Nov 02 20:15:21.288712 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.288639    5224 scope.go:122] "RemoveContainer" containerID="242d5f0582a33e336839df1a76ea445fb8adef906bdf561f797218b00cb2c2d6"
Nov 02 20:15:21.290439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.290416    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"
Nov 02 20:15:21.290439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.290423    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx" event={"ID":"2189a356-2417-4635-b1b4-22f3920a5f21","Type":"ContainerDied","Data":"1ca314dbff88ebb8da6fabce074e058a4c0d76b28e1474fa5815a406d0f1fd5e"}
Nov 02 20:15:21.292542 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.292524    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"
Nov 02 20:15:21.292655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.292556    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm" event={"ID":"ed59494c-0a34-41ab-ba57-ae829d35f753","Type":"ContainerDied","Data":"c24d4b8c708d18d59891cbe3801120631c11421ef05824e742a2c9e12e963dc6"}
Nov 02 20:15:21.303684 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.303664    5224 scope.go:122] "RemoveContainer" containerID="60d4606e9d0c0f103889f12f022228744b7cd422fb440f3f11df03f998b6b4c2"
Nov 02 20:15:21.313382 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.313360    5224 scope.go:122] "RemoveContainer" containerID="854fad10faf99e5761b29ed32d7682155b38867a4b5f97a61be161d7f9a921e3"
Nov 02 20:15:21.463902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.463819    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"]
Nov 02 20:15:21.468297 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.468265    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-9zws2"]
Nov 02 20:15:21.579663 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.579622    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"]
Nov 02 20:15:21.583910 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.583889    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-15203f-29894175-bfzbx"]
Nov 02 20:15:21.697420 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.697390    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"]
Nov 02 20:15:21.703049 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:21.703023    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-671e7c-29894175-sf5sm"]
Nov 02 20:15:22.134569 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.134538    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:15:22.212456 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.212422    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-certs\") pod \"baf37ca7-a400-495d-bce4-cf6613da5355\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") "
Nov 02 20:15:22.212655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.212469    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-data\") pod \"baf37ca7-a400-495d-bce4-cf6613da5355\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") "
Nov 02 20:15:22.212655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.212493    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-socket\") pod \"baf37ca7-a400-495d-bce4-cf6613da5355\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") "
Nov 02 20:15:22.212655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.212544    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-envoy\") pod \"baf37ca7-a400-495d-bce4-cf6613da5355\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") "
Nov 02 20:15:22.212655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.212592    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf37ca7-a400-495d-bce4-cf6613da5355-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf37ca7-a400-495d-bce4-cf6613da5355-istiod-ca-cert\") pod \"baf37ca7-a400-495d-bce4-cf6613da5355\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") "
Nov 02 20:15:22.212655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.212654    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-kube-api-access-25tcm\" (UniqueName: \"kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-kube-api-access-25tcm\") pod \"baf37ca7-a400-495d-bce4-cf6613da5355\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") "
Nov 02 20:15:22.212980 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.212753    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baf37ca7-a400-495d-bce4-cf6613da5355-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf37ca7-a400-495d-bce4-cf6613da5355-istio-podinfo\") pod \"baf37ca7-a400-495d-bce4-cf6613da5355\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") "
Nov 02 20:15:22.212980 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.212786    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-data" pod "baf37ca7-a400-495d-bce4-cf6613da5355" (UID: "baf37ca7-a400-495d-bce4-cf6613da5355"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:22.212980 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.212825    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-certs" pod "baf37ca7-a400-495d-bce4-cf6613da5355" (UID: "baf37ca7-a400-495d-bce4-cf6613da5355"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:22.212980 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.212831    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-istio-token\" (UniqueName: \"kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-istio-token\") pod \"baf37ca7-a400-495d-bce4-cf6613da5355\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") "
Nov 02 20:15:22.212980 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.212909    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-socket" pod "baf37ca7-a400-495d-bce4-cf6613da5355" (UID: "baf37ca7-a400-495d-bce4-cf6613da5355"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:22.212980 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.212978    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-credential-socket\") pod \"baf37ca7-a400-495d-bce4-cf6613da5355\" (UID: \"baf37ca7-a400-495d-bce4-cf6613da5355\") "
Nov 02 20:15:22.213297 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.213071    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf37ca7-a400-495d-bce4-cf6613da5355-istiod-ca-cert" pod "baf37ca7-a400-495d-bce4-cf6613da5355" (UID: "baf37ca7-a400-495d-bce4-cf6613da5355"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:22.213297 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.213220    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-credential-socket" pod "baf37ca7-a400-495d-bce4-cf6613da5355" (UID: "baf37ca7-a400-495d-bce4-cf6613da5355"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:22.213788 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.213738    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:22.213901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.213799    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:22.213901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.213818    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:22.213901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.213834    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf37ca7-a400-495d-bce4-cf6613da5355-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:22.213901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.213848    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:22.214993 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.214968    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baf37ca7-a400-495d-bce4-cf6613da5355-istio-podinfo" pod "baf37ca7-a400-495d-bce4-cf6613da5355" (UID: "baf37ca7-a400-495d-bce4-cf6613da5355"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:15:22.215070 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.215003    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-envoy" pod "baf37ca7-a400-495d-bce4-cf6613da5355" (UID: "baf37ca7-a400-495d-bce4-cf6613da5355"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:22.215070 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.215034    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-kube-api-access-25tcm" pod "baf37ca7-a400-495d-bce4-cf6613da5355" (UID: "baf37ca7-a400-495d-bce4-cf6613da5355"). InnerVolumeSpecName "kube-api-access-25tcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:22.215149 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.215119    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-istio-token" pod "baf37ca7-a400-495d-bce4-cf6613da5355" (UID: "baf37ca7-a400-495d-bce4-cf6613da5355"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:22.302413 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.302313    5224 generic.go:356] "Generic (PLEG): container finished" podID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerID="1c1f67f6f0d9510a4ada06d708f1c9b49b17fe13282f8bb1355d9419a8a0e696" exitCode=0
Nov 02 20:15:22.302413 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.302383    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6" event={"ID":"df701f2f-911b-40e0-9e88-a66aa003ee45","Type":"ContainerDied","Data":"1c1f67f6f0d9510a4ada06d708f1c9b49b17fe13282f8bb1355d9419a8a0e696"}
Nov 02 20:15:22.302942 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.302920    5224 scope.go:122] "RemoveContainer" containerID="1c1f67f6f0d9510a4ada06d708f1c9b49b17fe13282f8bb1355d9419a8a0e696"
Nov 02 20:15:22.304663 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.304628    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:15:22.304760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.304723    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:15:22.304841 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.304790    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:15:22.308639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.308501    5224 generic.go:356] "Generic (PLEG): container finished" podID="baf37ca7-a400-495d-bce4-cf6613da5355" containerID="28da88f973b102a70d58af30b5581b14df16bd78b11b377d4733a9979bf82600" exitCode=0
Nov 02 20:15:22.308639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.308580    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" event={"ID":"baf37ca7-a400-495d-bce4-cf6613da5355","Type":"ContainerDied","Data":"28da88f973b102a70d58af30b5581b14df16bd78b11b377d4733a9979bf82600"}
Nov 02 20:15:22.308639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.308626    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76" event={"ID":"baf37ca7-a400-495d-bce4-cf6613da5355","Type":"ContainerDied","Data":"691f873bbe2af65f4054292fb5d1dd45d7a5a3df82c5f20d8dc3db8414567a0f"}
Nov 02 20:15:22.308843 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.308659    5224 scope.go:122] "RemoveContainer" containerID="28da88f973b102a70d58af30b5581b14df16bd78b11b377d4733a9979bf82600"
Nov 02 20:15:22.308843 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.308802    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"
Nov 02 20:15:22.315277 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.315253    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf37ca7-a400-495d-bce4-cf6613da5355-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:22.315400 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.315277    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25tcm\" (UniqueName: \"kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-kube-api-access-25tcm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:22.315400 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.315295    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf37ca7-a400-495d-bce4-cf6613da5355-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:22.315400 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.315308    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf37ca7-a400-495d-bce4-cf6613da5355-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:22.323881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.323852    5224 scope.go:122] "RemoveContainer" containerID="db602aede4931e47a7f47abb2c97bb5bf89b56232a5521a9c5290fd681d328bb"
Nov 02 20:15:22.336366 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.336339    5224 scope.go:122] "RemoveContainer" containerID="28da88f973b102a70d58af30b5581b14df16bd78b11b377d4733a9979bf82600"
Nov 02 20:15:22.336836 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:22.336756    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28da88f973b102a70d58af30b5581b14df16bd78b11b377d4733a9979bf82600\": container with ID starting with 28da88f973b102a70d58af30b5581b14df16bd78b11b377d4733a9979bf82600 not found: ID does not exist" containerID="28da88f973b102a70d58af30b5581b14df16bd78b11b377d4733a9979bf82600"
Nov 02 20:15:22.336926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.336849    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28da88f973b102a70d58af30b5581b14df16bd78b11b377d4733a9979bf82600"} err="failed to get container status \"28da88f973b102a70d58af30b5581b14df16bd78b11b377d4733a9979bf82600\": rpc error: code = NotFound desc = could not find container \"28da88f973b102a70d58af30b5581b14df16bd78b11b377d4733a9979bf82600\": container with ID starting with 28da88f973b102a70d58af30b5581b14df16bd78b11b377d4733a9979bf82600 not found: ID does not exist"
Nov 02 20:15:22.336926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.336877    5224 scope.go:122] "RemoveContainer" containerID="db602aede4931e47a7f47abb2c97bb5bf89b56232a5521a9c5290fd681d328bb"
Nov 02 20:15:22.337454 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:22.337430    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db602aede4931e47a7f47abb2c97bb5bf89b56232a5521a9c5290fd681d328bb\": container with ID starting with db602aede4931e47a7f47abb2c97bb5bf89b56232a5521a9c5290fd681d328bb not found: ID does not exist" containerID="db602aede4931e47a7f47abb2c97bb5bf89b56232a5521a9c5290fd681d328bb"
Nov 02 20:15:22.337528 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.337466    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db602aede4931e47a7f47abb2c97bb5bf89b56232a5521a9c5290fd681d328bb"} err="failed to get container status \"db602aede4931e47a7f47abb2c97bb5bf89b56232a5521a9c5290fd681d328bb\": rpc error: code = NotFound desc = could not find container \"db602aede4931e47a7f47abb2c97bb5bf89b56232a5521a9c5290fd681d328bb\": container with ID starting with db602aede4931e47a7f47abb2c97bb5bf89b56232a5521a9c5290fd681d328bb not found: ID does not exist"
Nov 02 20:15:22.537605 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.537571    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"]
Nov 02 20:15:22.540716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:22.540687    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-45c182-openshift-default-6cd48c77f5-n8v76"]
Nov 02 20:15:23.187853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:23.186942    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2189a356-2417-4635-b1b4-22f3920a5f21" path="/var/lib/kubelet/pods/2189a356-2417-4635-b1b4-22f3920a5f21/volumes"
Nov 02 20:15:23.187853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:23.187801    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" path="/var/lib/kubelet/pods/baf37ca7-a400-495d-bce4-cf6613da5355/volumes"
Nov 02 20:15:23.188389 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:23.188359    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a" path="/var/lib/kubelet/pods/dbf16df4-48e0-417b-8a03-82cf8fb4a10a/volumes"
Nov 02 20:15:23.189375 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:23.189353    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed59494c-0a34-41ab-ba57-ae829d35f753" path="/var/lib/kubelet/pods/ed59494c-0a34-41ab-ba57-ae829d35f753/volumes"
Nov 02 20:15:23.281547 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:23.281489    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:23.318570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:23.318537    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6" event={"ID":"df701f2f-911b-40e0-9e88-a66aa003ee45","Type":"ContainerStarted","Data":"de9f36f0013b8d68b5b457b3fc9c5c10765b99f884513c4751396d34d7550def"}
Nov 02 20:15:23.478851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:23.478339    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:23.674157 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:23.674114    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:15:23.674356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:23.674261    5224 patch_prober.go:29] interesting pod/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 20:15:23.674356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:23.674310    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6" podUID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 20:15:24.234156 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:24.234100    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:24.469386 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:24.469347    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:24.677793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:24.676429    5224 patch_prober.go:29] interesting pod/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 20:15:24.677793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:24.676500    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6" podUID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 20:15:25.674240 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:25.674195    5224 patch_prober.go:29] interesting pod/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 20:15:25.674812 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:25.674265    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6" podUID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 20:15:25.911222 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:25.911169    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:26.403126 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:26.403088    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:26.593793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:26.576653    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:26.674850 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:26.674195    5224 patch_prober.go:29] interesting pod/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 20:15:26.674850 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:26.674258    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6" podUID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 20:15:27.207460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:27.207418    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:27.395474 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:27.395431    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:15:27.674204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:27.674152    5224 patch_prober.go:29] interesting pod/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 20:15:27.674426 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:27.674227    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6" podUID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 20:15:28.680205 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:28.680159    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:15:28.680875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:28.680606    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:15:28.681468 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:28.681444    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:15:29.760585 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:29.758516    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:30.972253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:30.972211    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:15:31.162915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:31.162877    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:32.448789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:32.448738    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"]
Nov 02 20:15:32.879876 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:32.879813    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:33.069426 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:33.069391    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:33.313447 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:33.313400    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:33.530606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:33.530567    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:33.706651 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:33.706552    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:33.844791 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:33.844664    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"]
Nov 02 20:15:33.908269 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:33.908228    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e04421b-rzchc"]
Nov 02 20:15:35.182529 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:35.182490    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd3729b-4162-427e-b40f-3d74e21b820e" path="/var/lib/kubelet/pods/fbd3729b-4162-427e-b40f-3d74e21b820e/volumes"
Nov 02 20:15:36.747922 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:36.747887    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"]
Nov 02 20:15:36.748380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:36.748123    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy" containerID="cri-o://9ba7033c857e6aea53a525b1c07f53d800e7edda3fe629fb0dea78adecb615f1" gracePeriod=30
Nov 02 20:15:37.467912 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:37.467852    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs" podUID="d9418ffd-ddec-4ef6-ae20-25f9b8875fc9" containerName="payload-processing" containerID="cri-o://9f3f823fdd3b4c0f4fb7caf695de6f1f9264e55226b737cc530d0facf040683e" gracePeriod=25
Nov 02 20:15:37.805859 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:37.805828    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"
Nov 02 20:15:37.819089 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:37.819031    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:37.898463 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:37.898420    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-kube-api-access-h6n82\" (UniqueName: \"kubernetes.io/projected/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-kube-api-access-h6n82\") pod \"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9\" (UID: \"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9\") "
Nov 02 20:15:37.898666 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:37.898508    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-plugins-config-volume\") pod \"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9\" (UID: \"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9\") "
Nov 02 20:15:37.898934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:37.898909    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-plugins-config-volume" pod "d9418ffd-ddec-4ef6-ae20-25f9b8875fc9" (UID: "d9418ffd-ddec-4ef6-ae20-25f9b8875fc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:37.899206 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:37.899178    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:37.900691 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:37.900669    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-kube-api-access-h6n82" pod "d9418ffd-ddec-4ef6-ae20-25f9b8875fc9" (UID: "d9418ffd-ddec-4ef6-ae20-25f9b8875fc9"). InnerVolumeSpecName "kube-api-access-h6n82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:38.000222 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.000181    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6n82\" (UniqueName: \"kubernetes.io/projected/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9-kube-api-access-h6n82\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:38.274106 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.274070    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"]
Nov 02 20:15:38.274307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.274282    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67" podUID="bb6de519-aa8e-4768-a96f-c930e7a14dad" containerName="istio-proxy" containerID="cri-o://42bcf710ade99afbe2cdca2241fef83d652f1fe4aba0345db71f90d27632676c" gracePeriod=30
Nov 02 20:15:38.325204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.325165    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"]
Nov 02 20:15:38.325412 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.325389    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7" podUID="b5d522fc-4998-4ef6-a473-f6fb90335551" containerName="istio-proxy" containerID="cri-o://a7b65e4d033faced083453664399e12b25a48699f3d7b5f52da5d3b527212c03" gracePeriod=30
Nov 02 20:15:38.524728 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.524689    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"]
Nov 02 20:15:38.525014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.524985    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf" podUID="0d8f8a81-0c49-4ad7-be08-71a5e5bde231" containerName="istio-proxy" containerID="cri-o://9ab848341bf106b9b4e7927d87fce9b5a39f0f87fc7e2042760957ddaf593cc1" gracePeriod=30
Nov 02 20:15:38.593112 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.593078    5224 generic.go:356] "Generic (PLEG): container finished" podID="d9418ffd-ddec-4ef6-ae20-25f9b8875fc9" containerID="9f3f823fdd3b4c0f4fb7caf695de6f1f9264e55226b737cc530d0facf040683e" exitCode=0
Nov 02 20:15:38.593315 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.593127    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs" event={"ID":"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9","Type":"ContainerDied","Data":"9f3f823fdd3b4c0f4fb7caf695de6f1f9264e55226b737cc530d0facf040683e"}
Nov 02 20:15:38.593315 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.593137    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"
Nov 02 20:15:38.593315 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.593173    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs" event={"ID":"d9418ffd-ddec-4ef6-ae20-25f9b8875fc9","Type":"ContainerDied","Data":"7db101222987a918134aa06e7805417b1a0b57034763c839f6014d6e887c638e"}
Nov 02 20:15:38.593315 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.593195    5224 scope.go:122] "RemoveContainer" containerID="9f3f823fdd3b4c0f4fb7caf695de6f1f9264e55226b737cc530d0facf040683e"
Nov 02 20:15:38.605472 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.605442    5224 scope.go:122] "RemoveContainer" containerID="9f3f823fdd3b4c0f4fb7caf695de6f1f9264e55226b737cc530d0facf040683e"
Nov 02 20:15:38.605925 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:38.605901    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3f823fdd3b4c0f4fb7caf695de6f1f9264e55226b737cc530d0facf040683e\": container with ID starting with 9f3f823fdd3b4c0f4fb7caf695de6f1f9264e55226b737cc530d0facf040683e not found: ID does not exist" containerID="9f3f823fdd3b4c0f4fb7caf695de6f1f9264e55226b737cc530d0facf040683e"
Nov 02 20:15:38.606060 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.605937    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3f823fdd3b4c0f4fb7caf695de6f1f9264e55226b737cc530d0facf040683e"} err="failed to get container status \"9f3f823fdd3b4c0f4fb7caf695de6f1f9264e55226b737cc530d0facf040683e\": rpc error: code = NotFound desc = could not find container \"9f3f823fdd3b4c0f4fb7caf695de6f1f9264e55226b737cc530d0facf040683e\": container with ID starting with 9f3f823fdd3b4c0f4fb7caf695de6f1f9264e55226b737cc530d0facf040683e not found: ID does not exist"
Nov 02 20:15:38.762212 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.762176    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"]
Nov 02 20:15:38.766756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.766718    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3021d-6d7c8dc9cf-48tqs"]
Nov 02 20:15:38.869614 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:38.869521    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:39.169381 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:39.169301    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9418ffd-ddec-4ef6-ae20-25f9b8875fc9" path="/var/lib/kubelet/pods/d9418ffd-ddec-4ef6-ae20-25f9b8875fc9/volumes"
Nov 02 20:15:39.326925 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:39.326889    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:39.566982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:39.566942    5224 scope.go:122] "RemoveContainer" containerID="4d9956fd6d2844da771881c98a618c9571ef3160fb163deb20d05b221150ec6a"
Nov 02 20:15:39.578413 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:39.578386    5224 scope.go:122] "RemoveContainer" containerID="a3a5876be327af15d0610114bd4e9cfb425d17862e06f3b1477e6b494b2fcb3d"
Nov 02 20:15:39.588509 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:39.588475    5224 scope.go:122] "RemoveContainer" containerID="f92f87eb3d49704cd22f42d4ae7f3d2f1a4f1320e14a05462c989369eef612d7"
Nov 02 20:15:39.607839 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:39.607803    5224 scope.go:122] "RemoveContainer" containerID="902dc2699b47cfcbc38496e7f908588be28b0e5641629a4f8d96ff19d3fbe244"
Nov 02 20:15:40.050406 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:40.050364    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"]
Nov 02 20:15:40.099739 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:40.099702    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5b11e8-t97kr"]
Nov 02 20:15:41.168866 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:41.168828    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18cc3e9f-b233-4e43-a40a-5e50bec1f680" path="/var/lib/kubelet/pods/18cc3e9f-b233-4e43-a40a-5e50bec1f680/volumes"
Nov 02 20:15:41.199974 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:41.199932    5224 patch_prober.go:29] interesting pod/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:15:41.200148 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:41.200008    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:41.364388 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:41.364349    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:41.565131 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:41.565084    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:42.018536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.018511    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:15:42.056525 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.056488    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istiod-ca-cert\") pod \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") "
Nov 02 20:15:42.056698 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.056542    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-certs\") pod \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") "
Nov 02 20:15:42.056698 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.056679    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-token\" (UniqueName: \"kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-token\") pod \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") "
Nov 02 20:15:42.056811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.056709    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-kube-api-access-x7br5\" (UniqueName: \"kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-kube-api-access-x7br5\") pod \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") "
Nov 02 20:15:42.056811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.056747    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-credential-socket\") pod \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") "
Nov 02 20:15:42.056893 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.056849    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-envoy\") pod \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") "
Nov 02 20:15:42.056893 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.056878    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-data\") pod \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") "
Nov 02 20:15:42.056992 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.056912    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-certs" pod "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" (UID: "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:42.056992 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.056940    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istiod-ca-cert" pod "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" (UID: "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:42.056992 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.056986    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-podinfo\") pod \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") "
Nov 02 20:15:42.057214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.057035    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-socket\") pod \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\" (UID: \"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece\") "
Nov 02 20:15:42.057214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.057136    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-credential-socket" pod "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" (UID: "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:42.057214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.057149    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-data" pod "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" (UID: "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:42.057446 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.057416    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-socket" pod "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" (UID: "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:42.057861 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.057837    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:42.057967 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.057866    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:42.057967 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.057886    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:42.057967 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.057902    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:42.057967 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.057917    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:42.059089 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.059062    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-envoy" pod "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" (UID: "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:42.059174 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.059062    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-kube-api-access-x7br5" pod "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" (UID: "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece"). InnerVolumeSpecName "kube-api-access-x7br5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:42.059174 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.059124    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-podinfo" pod "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" (UID: "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:15:42.059174 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.059150    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-token" pod "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" (UID: "17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:42.159536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.159439    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:42.159536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.159477    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7br5\" (UniqueName: \"kubernetes.io/projected/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-kube-api-access-x7br5\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:42.159536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.159488    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:42.159536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.159497    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:42.641245 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.641212    5224 generic.go:356] "Generic (PLEG): container finished" podID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerID="9ba7033c857e6aea53a525b1c07f53d800e7edda3fe629fb0dea78adecb615f1" exitCode=0
Nov 02 20:15:42.641726 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.641260    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"
Nov 02 20:15:42.641726 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.641301    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" event={"ID":"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece","Type":"ContainerDied","Data":"9ba7033c857e6aea53a525b1c07f53d800e7edda3fe629fb0dea78adecb615f1"}
Nov 02 20:15:42.641726 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.641356    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2" event={"ID":"17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece","Type":"ContainerDied","Data":"2842d742af37b5f957a63691cfb8efbde4bed4e4df39aba7107ec661c0d1cfb4"}
Nov 02 20:15:42.641726 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.641383    5224 scope.go:122] "RemoveContainer" containerID="9ba7033c857e6aea53a525b1c07f53d800e7edda3fe629fb0dea78adecb615f1"
Nov 02 20:15:42.643110 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.643085    5224 generic.go:356] "Generic (PLEG): container finished" podID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerID="ed5c10f39f9918a7a23cf82917b7196d5c7960eb0a164ad738b4e6d962f450a8" exitCode=0
Nov 02 20:15:42.643221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.643155    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" event={"ID":"40c3b61e-5330-4cab-8e14-235fefda4c8c","Type":"ContainerDied","Data":"ed5c10f39f9918a7a23cf82917b7196d5c7960eb0a164ad738b4e6d962f450a8"}
Nov 02 20:15:42.643592 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.643577    5224 scope.go:122] "RemoveContainer" containerID="ed5c10f39f9918a7a23cf82917b7196d5c7960eb0a164ad738b4e6d962f450a8"
Nov 02 20:15:42.645341 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.645310    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:15:42.645424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.645407    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:15:42.645477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.645462    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:15:42.652603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.652586    5224 scope.go:122] "RemoveContainer" containerID="9ba7033c857e6aea53a525b1c07f53d800e7edda3fe629fb0dea78adecb615f1"
Nov 02 20:15:42.653004 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:42.652978    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba7033c857e6aea53a525b1c07f53d800e7edda3fe629fb0dea78adecb615f1\": container with ID starting with 9ba7033c857e6aea53a525b1c07f53d800e7edda3fe629fb0dea78adecb615f1 not found: ID does not exist" containerID="9ba7033c857e6aea53a525b1c07f53d800e7edda3fe629fb0dea78adecb615f1"
Nov 02 20:15:42.653076 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.653010    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba7033c857e6aea53a525b1c07f53d800e7edda3fe629fb0dea78adecb615f1"} err="failed to get container status \"9ba7033c857e6aea53a525b1c07f53d800e7edda3fe629fb0dea78adecb615f1\": rpc error: code = NotFound desc = could not find container \"9ba7033c857e6aea53a525b1c07f53d800e7edda3fe629fb0dea78adecb615f1\": container with ID starting with 9ba7033c857e6aea53a525b1c07f53d800e7edda3fe629fb0dea78adecb615f1 not found: ID does not exist"
Nov 02 20:15:42.811541 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.811504    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"]
Nov 02 20:15:42.821421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:42.821386    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-f4xf2"]
Nov 02 20:15:43.177004 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.176966    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" path="/var/lib/kubelet/pods/17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece/volumes"
Nov 02 20:15:43.582409 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.581700    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 20:15:43.582409 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.581761    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7" podUID="b5d522fc-4998-4ef6-a473-f6fb90335551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 20:15:43.591009 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.590856    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:43.729849 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.727793    5224 generic.go:356] "Generic (PLEG): container finished" podID="0d8f8a81-0c49-4ad7-be08-71a5e5bde231" containerID="9ab848341bf106b9b4e7927d87fce9b5a39f0f87fc7e2042760957ddaf593cc1" exitCode=0
Nov 02 20:15:43.729849 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.727995    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf" event={"ID":"0d8f8a81-0c49-4ad7-be08-71a5e5bde231","Type":"ContainerDied","Data":"9ab848341bf106b9b4e7927d87fce9b5a39f0f87fc7e2042760957ddaf593cc1"}
Nov 02 20:15:43.766086 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.763177    5224 generic.go:356] "Generic (PLEG): container finished" podID="bb6de519-aa8e-4768-a96f-c930e7a14dad" containerID="42bcf710ade99afbe2cdca2241fef83d652f1fe4aba0345db71f90d27632676c" exitCode=0
Nov 02 20:15:43.766086 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.763350    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67" event={"ID":"bb6de519-aa8e-4768-a96f-c930e7a14dad","Type":"ContainerDied","Data":"42bcf710ade99afbe2cdca2241fef83d652f1fe4aba0345db71f90d27632676c"}
Nov 02 20:15:43.794371 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.782017    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" event={"ID":"40c3b61e-5330-4cab-8e14-235fefda4c8c","Type":"ContainerStarted","Data":"b7c99964278afee653b061d579b2e8ab5bb44ae74e8373b6e3f19c9827551874"}
Nov 02 20:15:43.811760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.811176    5224 generic.go:356] "Generic (PLEG): container finished" podID="b5d522fc-4998-4ef6-a473-f6fb90335551" containerID="a7b65e4d033faced083453664399e12b25a48699f3d7b5f52da5d3b527212c03" exitCode=0
Nov 02 20:15:43.811760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.811263    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7" event={"ID":"b5d522fc-4998-4ef6-a473-f6fb90335551","Type":"ContainerDied","Data":"a7b65e4d033faced083453664399e12b25a48699f3d7b5f52da5d3b527212c03"}
Nov 02 20:15:43.814074 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.813538    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:15:43.816502 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.815570    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:43.885793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.880406    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930075    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-podinfo\") pod \"b5d522fc-4998-4ef6-a473-f6fb90335551\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930190    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-certs\") pod \"bb6de519-aa8e-4768-a96f-c930e7a14dad\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930224    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-kube-api-access-k44dt\" (UniqueName: \"kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-kube-api-access-k44dt\") pod \"bb6de519-aa8e-4768-a96f-c930e7a14dad\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930279    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-envoy\") pod \"b5d522fc-4998-4ef6-a473-f6fb90335551\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930316    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-credential-socket\") pod \"b5d522fc-4998-4ef6-a473-f6fb90335551\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930355    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-podinfo\") pod \"bb6de519-aa8e-4768-a96f-c930e7a14dad\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930385    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-token\" (UniqueName: \"kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-token\") pod \"b5d522fc-4998-4ef6-a473-f6fb90335551\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930411    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-envoy\") pod \"bb6de519-aa8e-4768-a96f-c930e7a14dad\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930480    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-data\") pod \"bb6de519-aa8e-4768-a96f-c930e7a14dad\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930536    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-certs\") pod \"b5d522fc-4998-4ef6-a473-f6fb90335551\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930601    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-socket\") pod \"b5d522fc-4998-4ef6-a473-f6fb90335551\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930689    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-kube-api-access-8mpzm\" (UniqueName: \"kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-kube-api-access-8mpzm\") pod \"b5d522fc-4998-4ef6-a473-f6fb90335551\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930718    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6de519-aa8e-4768-a96f-c930e7a14dad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6de519-aa8e-4768-a96f-c930e7a14dad-istiod-ca-cert\") pod \"bb6de519-aa8e-4768-a96f-c930e7a14dad\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930757    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-token\" (UniqueName: \"kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-token\") pod \"bb6de519-aa8e-4768-a96f-c930e7a14dad\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930805    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-credential-socket\") pod \"bb6de519-aa8e-4768-a96f-c930e7a14dad\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930889    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-data\") pod \"b5d522fc-4998-4ef6-a473-f6fb90335551\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930935    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-socket\") pod \"bb6de519-aa8e-4768-a96f-c930e7a14dad\" (UID: \"bb6de519-aa8e-4768-a96f-c930e7a14dad\") "
Nov 02 20:15:43.931351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.930969    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d522fc-4998-4ef6-a473-f6fb90335551-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d522fc-4998-4ef6-a473-f6fb90335551-istiod-ca-cert\") pod \"b5d522fc-4998-4ef6-a473-f6fb90335551\" (UID: \"b5d522fc-4998-4ef6-a473-f6fb90335551\") "
Nov 02 20:15:43.933756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.933127    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-data" pod "bb6de519-aa8e-4768-a96f-c930e7a14dad" (UID: "bb6de519-aa8e-4768-a96f-c930e7a14dad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:43.933756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.933327    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6de519-aa8e-4768-a96f-c930e7a14dad-istiod-ca-cert" pod "bb6de519-aa8e-4768-a96f-c930e7a14dad" (UID: "bb6de519-aa8e-4768-a96f-c930e7a14dad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:43.933756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.933648    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d522fc-4998-4ef6-a473-f6fb90335551-istiod-ca-cert" pod "b5d522fc-4998-4ef6-a473-f6fb90335551" (UID: "b5d522fc-4998-4ef6-a473-f6fb90335551"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:43.937115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.934181    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-credential-socket" pod "b5d522fc-4998-4ef6-a473-f6fb90335551" (UID: "b5d522fc-4998-4ef6-a473-f6fb90335551"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:43.937115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.934156    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-credential-socket" pod "bb6de519-aa8e-4768-a96f-c930e7a14dad" (UID: "bb6de519-aa8e-4768-a96f-c930e7a14dad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:43.937115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.934572    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-certs" pod "bb6de519-aa8e-4768-a96f-c930e7a14dad" (UID: "bb6de519-aa8e-4768-a96f-c930e7a14dad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:43.937115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.934681    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-certs" pod "b5d522fc-4998-4ef6-a473-f6fb90335551" (UID: "b5d522fc-4998-4ef6-a473-f6fb90335551"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:43.937115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.936936    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-data" pod "b5d522fc-4998-4ef6-a473-f6fb90335551" (UID: "b5d522fc-4998-4ef6-a473-f6fb90335551"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:43.946601 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.937358    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-socket" pod "bb6de519-aa8e-4768-a96f-c930e7a14dad" (UID: "bb6de519-aa8e-4768-a96f-c930e7a14dad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:43.946601 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.937842    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-socket" pod "b5d522fc-4998-4ef6-a473-f6fb90335551" (UID: "b5d522fc-4998-4ef6-a473-f6fb90335551"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:43.981450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.946901    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-kube-api-access-8mpzm" pod "b5d522fc-4998-4ef6-a473-f6fb90335551" (UID: "b5d522fc-4998-4ef6-a473-f6fb90335551"). InnerVolumeSpecName "kube-api-access-8mpzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:43.981450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.951668    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-token" pod "bb6de519-aa8e-4768-a96f-c930e7a14dad" (UID: "bb6de519-aa8e-4768-a96f-c930e7a14dad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:43.981450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.953725    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-podinfo" pod "b5d522fc-4998-4ef6-a473-f6fb90335551" (UID: "b5d522fc-4998-4ef6-a473-f6fb90335551"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:15:43.981450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.954434    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-token" pod "b5d522fc-4998-4ef6-a473-f6fb90335551" (UID: "b5d522fc-4998-4ef6-a473-f6fb90335551"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:43.981450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.954538    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-podinfo" pod "bb6de519-aa8e-4768-a96f-c930e7a14dad" (UID: "bb6de519-aa8e-4768-a96f-c930e7a14dad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:15:43.981450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.954914    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-envoy" pod "b5d522fc-4998-4ef6-a473-f6fb90335551" (UID: "b5d522fc-4998-4ef6-a473-f6fb90335551"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:43.981450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.955290    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-envoy" pod "bb6de519-aa8e-4768-a96f-c930e7a14dad" (UID: "bb6de519-aa8e-4768-a96f-c930e7a14dad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:43.993794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:43.993726    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-kube-api-access-k44dt" pod "bb6de519-aa8e-4768-a96f-c930e7a14dad" (UID: "bb6de519-aa8e-4768-a96f-c930e7a14dad"). InnerVolumeSpecName "kube-api-access-k44dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:44.033937 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.033844    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.033937 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.033897    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.033937 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.033918    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.033937 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.033935    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mpzm\" (UniqueName: \"kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-kube-api-access-8mpzm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.033950    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6de519-aa8e-4768-a96f-c930e7a14dad-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.033963    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.033978    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.033991    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.034006    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.034020    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d522fc-4998-4ef6-a473-f6fb90335551-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.034033    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.034046    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.034059    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k44dt\" (UniqueName: \"kubernetes.io/projected/bb6de519-aa8e-4768-a96f-c930e7a14dad-kube-api-access-k44dt\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.034073    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.034086    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d522fc-4998-4ef6-a473-f6fb90335551-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.034104    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.034120    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d522fc-4998-4ef6-a473-f6fb90335551-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.034319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.034133    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6de519-aa8e-4768-a96f-c930e7a14dad-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.169803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.169187    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:15:44.248296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.245949    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-token\" (UniqueName: \"kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-token\") pod \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") "
Nov 02 20:15:44.248296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.246005    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istiod-ca-cert\") pod \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") "
Nov 02 20:15:44.248296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.246030    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-podinfo\") pod \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") "
Nov 02 20:15:44.248296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.246104    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-credential-socket\") pod \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") "
Nov 02 20:15:44.248296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.246153    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-data\") pod \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") "
Nov 02 20:15:44.248296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.246178    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-envoy\") pod \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") "
Nov 02 20:15:44.248296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.246219    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-kube-api-access-7dvz5\" (UniqueName: \"kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-kube-api-access-7dvz5\") pod \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") "
Nov 02 20:15:44.248296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.246240    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-certs\") pod \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") "
Nov 02 20:15:44.248296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.246315    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-socket\") pod \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\" (UID: \"0d8f8a81-0c49-4ad7-be08-71a5e5bde231\") "
Nov 02 20:15:44.248883 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.248440    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istiod-ca-cert" pod "0d8f8a81-0c49-4ad7-be08-71a5e5bde231" (UID: "0d8f8a81-0c49-4ad7-be08-71a5e5bde231"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:44.248883 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.248500    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-certs" pod "0d8f8a81-0c49-4ad7-be08-71a5e5bde231" (UID: "0d8f8a81-0c49-4ad7-be08-71a5e5bde231"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:44.248883 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.248698    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-socket" pod "0d8f8a81-0c49-4ad7-be08-71a5e5bde231" (UID: "0d8f8a81-0c49-4ad7-be08-71a5e5bde231"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:44.249455 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.249415    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-envoy" pod "0d8f8a81-0c49-4ad7-be08-71a5e5bde231" (UID: "0d8f8a81-0c49-4ad7-be08-71a5e5bde231"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:44.250160 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.250090    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-kube-api-access-7dvz5" pod "0d8f8a81-0c49-4ad7-be08-71a5e5bde231" (UID: "0d8f8a81-0c49-4ad7-be08-71a5e5bde231"). InnerVolumeSpecName "kube-api-access-7dvz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:44.251918 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.251893    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-credential-socket" pod "0d8f8a81-0c49-4ad7-be08-71a5e5bde231" (UID: "0d8f8a81-0c49-4ad7-be08-71a5e5bde231"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:44.253506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.253469    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-token" pod "0d8f8a81-0c49-4ad7-be08-71a5e5bde231" (UID: "0d8f8a81-0c49-4ad7-be08-71a5e5bde231"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:44.261797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.261287    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:15:44.261797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.261416    5224 patch_prober.go:29] interesting pod/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 20:15:44.261797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.261459    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 20:15:44.273832 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.267010    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-podinfo" pod "0d8f8a81-0c49-4ad7-be08-71a5e5bde231" (UID: "0d8f8a81-0c49-4ad7-be08-71a5e5bde231"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:15:44.274037 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.273897    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-data" pod "0d8f8a81-0c49-4ad7-be08-71a5e5bde231" (UID: "0d8f8a81-0c49-4ad7-be08-71a5e5bde231"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:15:44.344041 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.343999    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:44.348130 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.348093    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.348130 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.348131    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.348392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.348143    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.348392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.348152    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dvz5\" (UniqueName: \"kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-kube-api-access-7dvz5\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.348392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.348164    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.348392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.348178    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.348392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.348187    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.348392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.348195    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.348392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.348203    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d8f8a81-0c49-4ad7-be08-71a5e5bde231-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:44.594749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.594714    5224 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:15:44.594950 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.594791    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67" podUID="bb6de519-aa8e-4768-a96f-c930e7a14dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:44.818744 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.818711    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"
Nov 02 20:15:44.819195 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.818720    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7" event={"ID":"b5d522fc-4998-4ef6-a473-f6fb90335551","Type":"ContainerDied","Data":"019dd87236604c73bdf32960a4870ebfcf81b229ee2335a922340440177f96a5"}
Nov 02 20:15:44.819195 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.818860    5224 scope.go:122] "RemoveContainer" containerID="a7b65e4d033faced083453664399e12b25a48699f3d7b5f52da5d3b527212c03"
Nov 02 20:15:44.820497 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.820464    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"
Nov 02 20:15:44.820592 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.820506    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf" event={"ID":"0d8f8a81-0c49-4ad7-be08-71a5e5bde231","Type":"ContainerDied","Data":"7c6c3b4b7284034c338d1ec28e0b61ae4613fcca3a15220b6e056907f27b0816"}
Nov 02 20:15:44.837783 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.822235    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"
Nov 02 20:15:44.837783 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.822251    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67" event={"ID":"bb6de519-aa8e-4768-a96f-c930e7a14dad","Type":"ContainerDied","Data":"5b554a524d189cb318db9c2da7c07092659d19510f553f9acafbe3bb3acdcc98"}
Nov 02 20:15:44.839487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.839465    5224 scope.go:122] "RemoveContainer" containerID="9ab848341bf106b9b4e7927d87fce9b5a39f0f87fc7e2042760957ddaf593cc1"
Nov 02 20:15:44.887021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.886944    5224 scope.go:122] "RemoveContainer" containerID="42bcf710ade99afbe2cdca2241fef83d652f1fe4aba0345db71f90d27632676c"
Nov 02 20:15:44.991982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:44.991950    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"]
Nov 02 20:15:45.042929 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:45.042888    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-822qf"]
Nov 02 20:15:45.105784 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:45.105731    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"]
Nov 02 20:15:45.110132 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:45.110109    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-xzt67"]
Nov 02 20:15:45.181577 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:45.181496    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d8f8a81-0c49-4ad7-be08-71a5e5bde231" path="/var/lib/kubelet/pods/0d8f8a81-0c49-4ad7-be08-71a5e5bde231/volumes"
Nov 02 20:15:45.182026 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:45.182012    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6de519-aa8e-4768-a96f-c930e7a14dad" path="/var/lib/kubelet/pods/bb6de519-aa8e-4768-a96f-c930e7a14dad/volumes"
Nov 02 20:15:45.224618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:45.224581    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"]
Nov 02 20:15:45.230232 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:45.230199    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-jhqv7"]
Nov 02 20:15:45.263794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:45.260231    5224 patch_prober.go:29] interesting pod/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 20:15:45.263794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:45.260295    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 20:15:46.273954 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:46.273913    5224 patch_prober.go:29] interesting pod/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 20:15:46.274397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:46.273972    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 20:15:47.180919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:47.180886    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d522fc-4998-4ef6-a473-f6fb90335551" path="/var/lib/kubelet/pods/b5d522fc-4998-4ef6-a473-f6fb90335551/volumes"
Nov 02 20:15:47.270793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:47.264168    5224 patch_prober.go:29] interesting pod/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 20:15:47.270793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:47.264236    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 20:15:48.261855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:48.259240    5224 patch_prober.go:29] interesting pod/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 20:15:48.261855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:48.259323    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 20:15:49.261733 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:49.261701    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:15:49.262185 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:49.262157    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:15:49.262914 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:49.262896    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:15:49.791573 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:49.791534    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:50.453262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:50.453223    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:51.156572 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:51.156527    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:51.898816 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:51.898746    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:52.150418 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:52.150326    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:53.801089 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:53.801035    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:54.652836 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:54.652797    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:55.175128 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:55.175088    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:56.215895 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:56.215857    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:56.407821 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:56.407786    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:57.359432 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:15:57.359402    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:15:58.297240 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:58.297204    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:15:59.370537 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:59.370500    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:15:59.546806 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:15:59.546449    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:00.537391 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:00.537333    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:06.046579 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:06.046535    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:09.100865 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:09.100763    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:11.469016 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:11.468971    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:11.642204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:11.642146    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:16.184192 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:16.184148    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:17.558007 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:17.557968    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:18.327969 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:18.327929    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:18.556104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:18.556061    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:18.936662 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:18.936621    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:19.146055 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:19.146013    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:19.941600 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:19.941559    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:20.500906 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:20.500861    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:23.123233 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:23.123193    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:23.650869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:23.650808    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:23.824820 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:23.824762    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:24.046803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:24.046751    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:27.361865 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:16:27.361822    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:16:28.528170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:28.528124    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:30.468073 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:30.468020    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:33.377612 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:33.377576    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:16:34.801678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:34.801637    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:36.657153 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:36.657093    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:37.120186 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:37.120142    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:37.305198 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:37.305163    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:37.832041 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:37.832001    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:38.027869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:38.027822    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:38.714985 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:38.714943    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:39.228170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:39.228133    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:39.730625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:39.730589    5224 scope.go:122] "RemoveContainer" containerID="5d9cc1133b38a2b821f00a558d1c45a0959adfa665c010cc8924682303dc2577"
Nov 02 20:16:39.741219 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:39.741190    5224 scope.go:122] "RemoveContainer" containerID="61a8ff520772e0a4e482d5cbbafa09f3009b2f3cbf4750b8b752f59b4e7c9f9e"
Nov 02 20:16:40.208679 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:40.208637    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:40.940705 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:40.940659    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:41.466757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:41.466718    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:42.095189 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:42.095153    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:47.080348 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:47.080302    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:47.265979 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:47.265924    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:48.421037 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:48.420996    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:48.596779 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:48.596721    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:52.250981 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:52.250942    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:52.483845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:52.483804    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:52.680517 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:52.680479    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:52.861159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:52.861080    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:53.039529 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:53.039491    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:53.290603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:53.290506    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:53.909861 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:53.909816    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:54.130375 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:54.130338    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:55.019491 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:55.019446    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:57.027340 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:57.027308    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:57.361274 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:16:57.361183    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:16:58.049791 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:58.041471    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:58.249078 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:58.249038    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:58.476934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:58.476840    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:59.032940 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:59.032821    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:59.251381 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:59.251340    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:16:59.716838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:16:59.716797    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:00.079794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:00.078850    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:01.847194 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:01.847157    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:02.025066 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:02.025025    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:03.688665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:03.688616    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:03.899351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:03.899312    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:04.637155 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:04.637099    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:06.824163 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:06.824121    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:07.001669 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:07.001627    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:07.918333 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:07.918292    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:08.132829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:08.132782    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:09.521549 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:09.521475    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:09.984434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:09.984309    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:09.991433 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:09.991398    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"]
Nov 02 20:17:09.991679 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:09.991652    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy" containerID="cri-o://b7c99964278afee653b061d579b2e8ab5bb44ae74e8373b6e3f19c9827551874" gracePeriod=30
Nov 02 20:17:10.167808 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:10.167751    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:11.254903 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:11.254863    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:11.878383 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:11.877196    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:12.143360 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:12.143251    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:15.254983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.254937    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:15.279620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.279481    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:17:15.353048 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353008    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c3b61e-5330-4cab-8e14-235fefda4c8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c3b61e-5330-4cab-8e14-235fefda4c8c-istiod-ca-cert\") pod \"40c3b61e-5330-4cab-8e14-235fefda4c8c\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") "
Nov 02 20:17:15.353275 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353099    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-data\") pod \"40c3b61e-5330-4cab-8e14-235fefda4c8c\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") "
Nov 02 20:17:15.353275 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353144    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-token\" (UniqueName: \"kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-token\") pod \"40c3b61e-5330-4cab-8e14-235fefda4c8c\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") "
Nov 02 20:17:15.353275 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353184    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-certs\") pod \"40c3b61e-5330-4cab-8e14-235fefda4c8c\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") "
Nov 02 20:17:15.353275 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353210    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-socket\") pod \"40c3b61e-5330-4cab-8e14-235fefda4c8c\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") "
Nov 02 20:17:15.353567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353307    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-envoy\") pod \"40c3b61e-5330-4cab-8e14-235fefda4c8c\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") "
Nov 02 20:17:15.353567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353402    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-credential-socket\") pod \"40c3b61e-5330-4cab-8e14-235fefda4c8c\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") "
Nov 02 20:17:15.353567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353431    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-podinfo\") pod \"40c3b61e-5330-4cab-8e14-235fefda4c8c\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") "
Nov 02 20:17:15.353567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353466    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-data" pod "40c3b61e-5330-4cab-8e14-235fefda4c8c" (UID: "40c3b61e-5330-4cab-8e14-235fefda4c8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:15.353760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353573    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c3b61e-5330-4cab-8e14-235fefda4c8c-istiod-ca-cert" pod "40c3b61e-5330-4cab-8e14-235fefda4c8c" (UID: "40c3b61e-5330-4cab-8e14-235fefda4c8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:17:15.353760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353585    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-certs" pod "40c3b61e-5330-4cab-8e14-235fefda4c8c" (UID: "40c3b61e-5330-4cab-8e14-235fefda4c8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:15.353760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353598    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-socket" pod "40c3b61e-5330-4cab-8e14-235fefda4c8c" (UID: "40c3b61e-5330-4cab-8e14-235fefda4c8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:15.353760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353630    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-kube-api-access-84g8t\" (UniqueName: \"kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-kube-api-access-84g8t\") pod \"40c3b61e-5330-4cab-8e14-235fefda4c8c\" (UID: \"40c3b61e-5330-4cab-8e14-235fefda4c8c\") "
Nov 02 20:17:15.353760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.353726    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-credential-socket" pod "40c3b61e-5330-4cab-8e14-235fefda4c8c" (UID: "40c3b61e-5330-4cab-8e14-235fefda4c8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:15.354156 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.354136    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:15.354223 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.354162    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c3b61e-5330-4cab-8e14-235fefda4c8c-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:15.354223 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.354176    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:15.354223 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.354191    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:15.354223 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.354207    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:15.356289 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.356262    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-token" pod "40c3b61e-5330-4cab-8e14-235fefda4c8c" (UID: "40c3b61e-5330-4cab-8e14-235fefda4c8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:15.356289 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.356260    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-podinfo" pod "40c3b61e-5330-4cab-8e14-235fefda4c8c" (UID: "40c3b61e-5330-4cab-8e14-235fefda4c8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:17:15.356289 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.356266    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-envoy" pod "40c3b61e-5330-4cab-8e14-235fefda4c8c" (UID: "40c3b61e-5330-4cab-8e14-235fefda4c8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:15.356497 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.356290    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-kube-api-access-84g8t" pod "40c3b61e-5330-4cab-8e14-235fefda4c8c" (UID: "40c3b61e-5330-4cab-8e14-235fefda4c8c"). InnerVolumeSpecName "kube-api-access-84g8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:15.455498 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.455463    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84g8t\" (UniqueName: \"kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-kube-api-access-84g8t\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:15.455498 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.455496    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:15.455498 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.455507    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:15.455755 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.455517    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c3b61e-5330-4cab-8e14-235fefda4c8c-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:15.538352 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.538204    5224 generic.go:356] "Generic (PLEG): container finished" podID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerID="b7c99964278afee653b061d579b2e8ab5bb44ae74e8373b6e3f19c9827551874" exitCode=0
Nov 02 20:17:15.538780 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.538387    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"
Nov 02 20:17:15.538780 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.538413    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" event={"ID":"40c3b61e-5330-4cab-8e14-235fefda4c8c","Type":"ContainerDied","Data":"b7c99964278afee653b061d579b2e8ab5bb44ae74e8373b6e3f19c9827551874"}
Nov 02 20:17:15.538780 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.538513    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x" event={"ID":"40c3b61e-5330-4cab-8e14-235fefda4c8c","Type":"ContainerDied","Data":"8d124966e542b44f32f3f945534b8841906b95189aceed8eed4ef539fb75cc84"}
Nov 02 20:17:15.538780 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.538560    5224 scope.go:122] "RemoveContainer" containerID="b7c99964278afee653b061d579b2e8ab5bb44ae74e8373b6e3f19c9827551874"
Nov 02 20:17:15.550017 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.549994    5224 scope.go:122] "RemoveContainer" containerID="ed5c10f39f9918a7a23cf82917b7196d5c7960eb0a164ad738b4e6d962f450a8"
Nov 02 20:17:15.559556 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.559536    5224 scope.go:122] "RemoveContainer" containerID="b7c99964278afee653b061d579b2e8ab5bb44ae74e8373b6e3f19c9827551874"
Nov 02 20:17:15.559925 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:17:15.559899    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c99964278afee653b061d579b2e8ab5bb44ae74e8373b6e3f19c9827551874\": container with ID starting with b7c99964278afee653b061d579b2e8ab5bb44ae74e8373b6e3f19c9827551874 not found: ID does not exist" containerID="b7c99964278afee653b061d579b2e8ab5bb44ae74e8373b6e3f19c9827551874"
Nov 02 20:17:15.560047 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.559939    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c99964278afee653b061d579b2e8ab5bb44ae74e8373b6e3f19c9827551874"} err="failed to get container status \"b7c99964278afee653b061d579b2e8ab5bb44ae74e8373b6e3f19c9827551874\": rpc error: code = NotFound desc = could not find container \"b7c99964278afee653b061d579b2e8ab5bb44ae74e8373b6e3f19c9827551874\": container with ID starting with b7c99964278afee653b061d579b2e8ab5bb44ae74e8373b6e3f19c9827551874 not found: ID does not exist"
Nov 02 20:17:15.560047 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.559966    5224 scope.go:122] "RemoveContainer" containerID="ed5c10f39f9918a7a23cf82917b7196d5c7960eb0a164ad738b4e6d962f450a8"
Nov 02 20:17:15.560278 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:17:15.560258    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5c10f39f9918a7a23cf82917b7196d5c7960eb0a164ad738b4e6d962f450a8\": container with ID starting with ed5c10f39f9918a7a23cf82917b7196d5c7960eb0a164ad738b4e6d962f450a8 not found: ID does not exist" containerID="ed5c10f39f9918a7a23cf82917b7196d5c7960eb0a164ad738b4e6d962f450a8"
Nov 02 20:17:15.560326 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.560287    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5c10f39f9918a7a23cf82917b7196d5c7960eb0a164ad738b4e6d962f450a8"} err="failed to get container status \"ed5c10f39f9918a7a23cf82917b7196d5c7960eb0a164ad738b4e6d962f450a8\": rpc error: code = NotFound desc = could not find container \"ed5c10f39f9918a7a23cf82917b7196d5c7960eb0a164ad738b4e6d962f450a8\": container with ID starting with ed5c10f39f9918a7a23cf82917b7196d5c7960eb0a164ad738b4e6d962f450a8 not found: ID does not exist"
Nov 02 20:17:15.706719 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.706687    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"]
Nov 02 20:17:15.710069 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:15.710045    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e0aa5c-openshift-default-74c5cbf98f-zcg8x"]
Nov 02 20:17:16.814955 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:16.814913    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:16.998977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:16.998916    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:17.168398 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:17.168296    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" path="/var/lib/kubelet/pods/40c3b61e-5330-4cab-8e14-235fefda4c8c/volumes"
Nov 02 20:17:17.192023 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:17.191975    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:20.093838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.093699    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:20.314950 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.314912    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.644741    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"]
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645472    5224 cpu_manager.go:414] "containerMap: removing container" podUID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a" containerName="cleanup"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645508    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a" containerName="cleanup"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645536    5224 cpu_manager.go:414] "containerMap: removing container" podUID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a" containerName="cleanup"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645547    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a" containerName="cleanup"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645564    5224 cpu_manager.go:414] "containerMap: removing container" podUID="56b298bc-073a-49e8-86ae-998b10102bfe" containerName="cleanup"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645575    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b298bc-073a-49e8-86ae-998b10102bfe" containerName="cleanup"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645592    5224 cpu_manager.go:414] "containerMap: removing container" podUID="95f31166-a27b-4a71-98d0-a727faea4589" containerName="cleanup"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645603    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f31166-a27b-4a71-98d0-a727faea4589" containerName="cleanup"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645624    5224 cpu_manager.go:414] "containerMap: removing container" podUID="b5d522fc-4998-4ef6-a473-f6fb90335551" containerName="istio-proxy"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645631    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d522fc-4998-4ef6-a473-f6fb90335551" containerName="istio-proxy"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645647    5224 cpu_manager.go:414] "containerMap: removing container" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398" containerName="cleanup"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645657    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398" containerName="cleanup"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645668    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d9418ffd-ddec-4ef6-ae20-25f9b8875fc9" containerName="payload-processing"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645676    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9418ffd-ddec-4ef6-ae20-25f9b8875fc9" containerName="payload-processing"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645696    5224 cpu_manager.go:414] "containerMap: removing container" podUID="bb6de519-aa8e-4768-a96f-c930e7a14dad" containerName="istio-proxy"
Nov 02 20:17:20.645872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.645703    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6de519-aa8e-4768-a96f-c930e7a14dad" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647043    5224 cpu_manager.go:414] "containerMap: removing container" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647069    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647081    5224 cpu_manager.go:414] "containerMap: removing container" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647089    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647113    5224 cpu_manager.go:414] "containerMap: removing container" podUID="56b298bc-073a-49e8-86ae-998b10102bfe" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647121    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b298bc-073a-49e8-86ae-998b10102bfe" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647134    5224 cpu_manager.go:414] "containerMap: removing container" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647143    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647158    5224 cpu_manager.go:414] "containerMap: removing container" podUID="2189a356-2417-4635-b1b4-22f3920a5f21" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647165    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2189a356-2417-4635-b1b4-22f3920a5f21" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647186    5224 cpu_manager.go:414] "containerMap: removing container" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647193    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647209    5224 cpu_manager.go:414] "containerMap: removing container" podUID="ed59494c-0a34-41ab-ba57-ae829d35f753" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647216    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed59494c-0a34-41ab-ba57-ae829d35f753" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647230    5224 cpu_manager.go:414] "containerMap: removing container" podUID="0d8f8a81-0c49-4ad7-be08-71a5e5bde231" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647237    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d8f8a81-0c49-4ad7-be08-71a5e5bde231" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647260    5224 cpu_manager.go:414] "containerMap: removing container" podUID="56b298bc-073a-49e8-86ae-998b10102bfe" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647273    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b298bc-073a-49e8-86ae-998b10102bfe" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647292    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647304    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647326    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647340    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647362    5224 cpu_manager.go:414] "containerMap: removing container" podUID="2189a356-2417-4635-b1b4-22f3920a5f21" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647373    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2189a356-2417-4635-b1b4-22f3920a5f21" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647393    5224 cpu_manager.go:414] "containerMap: removing container" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647401    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647412    5224 cpu_manager.go:414] "containerMap: removing container" podUID="21980784-069f-41e3-ae48-40cd1c32186c" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647420    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21980784-069f-41e3-ae48-40cd1c32186c" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647431    5224 cpu_manager.go:414] "containerMap: removing container" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647439    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647465    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647472    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647487    5224 cpu_manager.go:414] "containerMap: removing container" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647495    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647507    5224 cpu_manager.go:414] "containerMap: removing container" podUID="ed59494c-0a34-41ab-ba57-ae829d35f753" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647514    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed59494c-0a34-41ab-ba57-ae829d35f753" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647528    5224 cpu_manager.go:414] "containerMap: removing container" podUID="ed59494c-0a34-41ab-ba57-ae829d35f753" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647536    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed59494c-0a34-41ab-ba57-ae829d35f753" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647547    5224 cpu_manager.go:414] "containerMap: removing container" podUID="95f31166-a27b-4a71-98d0-a727faea4589" containerName="cleanup"
Nov 02 20:17:20.647970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.647609    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f31166-a27b-4a71-98d0-a727faea4589" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648784    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648811    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b298bc-073a-49e8-86ae-998b10102bfe" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648827    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" containerName="istio-proxy"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648843    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="2189a356-2417-4635-b1b4-22f3920a5f21" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648854    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf37ca7-a400-495d-bce4-cf6613da5355" containerName="istio-proxy"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648866    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed59494c-0a34-41ab-ba57-ae829d35f753" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648876    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d8f8a81-0c49-4ad7-be08-71a5e5bde231" containerName="istio-proxy"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648887    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b298bc-073a-49e8-86ae-998b10102bfe" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648897    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648908    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f31166-a27b-4a71-98d0-a727faea4589" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648917    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648927    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="2189a356-2417-4635-b1b4-22f3920a5f21" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648936    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f0d2b1-4f9d-41f1-94a4-c7903a2d6ece" containerName="istio-proxy"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648948    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="21980784-069f-41e3-ae48-40cd1c32186c" containerName="istio-proxy"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648960    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648968    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed59494c-0a34-41ab-ba57-ae829d35f753" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648979    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed59494c-0a34-41ab-ba57-ae829d35f753" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.648991    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f31166-a27b-4a71-98d0-a727faea4589" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.649001    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.649010    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.649019    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.649029    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f31166-a27b-4a71-98d0-a727faea4589" containerName="cleanup"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.649041    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d522fc-4998-4ef6-a473-f6fb90335551" containerName="istio-proxy"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.649051    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9418ffd-ddec-4ef6-ae20-25f9b8875fc9" containerName="payload-processing"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.649063    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6de519-aa8e-4768-a96f-c930e7a14dad" containerName="istio-proxy"
Nov 02 20:17:20.655270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.649074    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398" containerName="cleanup"
Nov 02 20:17:20.664157 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.664122    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"
Nov 02 20:17:20.695847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.695799    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"]
Nov 02 20:17:20.738490 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.738447    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq9jx\" (UniqueName: \"kubernetes.io/projected/05c7506a-5bc2-4373-8f99-644b72d4c65a-kube-api-access-hq9jx\") pod \"payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs\" (UID: \"05c7506a-5bc2-4373-8f99-644b72d4c65a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"
Nov 02 20:17:20.738687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.738583    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c7506a-5bc2-4373-8f99-644b72d4c65a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs\" (UID: \"05c7506a-5bc2-4373-8f99-644b72d4c65a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"
Nov 02 20:17:20.744064 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.744038    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-296ae9\""
Nov 02 20:17:20.744133 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.744042    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-296ae9-dockercfg-h6xwk\""
Nov 02 20:17:20.780525 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.780481    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:20.840076 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.840043    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq9jx\" (UniqueName: \"kubernetes.io/projected/05c7506a-5bc2-4373-8f99-644b72d4c65a-kube-api-access-hq9jx\") pod \"payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs\" (UID: \"05c7506a-5bc2-4373-8f99-644b72d4c65a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"
Nov 02 20:17:20.840291 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.840137    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c7506a-5bc2-4373-8f99-644b72d4c65a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs\" (UID: \"05c7506a-5bc2-4373-8f99-644b72d4c65a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"
Nov 02 20:17:20.841013 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.840991    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c7506a-5bc2-4373-8f99-644b72d4c65a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs\" (UID: \"05c7506a-5bc2-4373-8f99-644b72d4c65a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"
Nov 02 20:17:20.898699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.898616    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq9jx\" (UniqueName: \"kubernetes.io/projected/05c7506a-5bc2-4373-8f99-644b72d4c65a-kube-api-access-hq9jx\") pod \"payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs\" (UID: \"05c7506a-5bc2-4373-8f99-644b72d4c65a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"
Nov 02 20:17:20.979462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:20.979414    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"
Nov 02 20:17:21.351607 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:17:21.351571    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c7506a_5bc2_4373_8f99_644b72d4c65a.slice/crio-687e1fda801e5728ffe3ec8d6744fd12d054177212ffb6b0f67c4d02a1e2470f WatchSource:0}: Error finding container 687e1fda801e5728ffe3ec8d6744fd12d054177212ffb6b0f67c4d02a1e2470f: Status 404 returned error can't find the container with id 687e1fda801e5728ffe3ec8d6744fd12d054177212ffb6b0f67c4d02a1e2470f
Nov 02 20:17:21.354423 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:21.354391    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"]
Nov 02 20:17:21.647839 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:21.647713    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs" event={"ID":"05c7506a-5bc2-4373-8f99-644b72d4c65a","Type":"ContainerStarted","Data":"d5cad01b2bf9ba5771ace176d8cc3af6ef681242bc5a0cbde4dcc532b132cc51"}
Nov 02 20:17:21.647839 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:21.647812    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs" event={"ID":"05c7506a-5bc2-4373-8f99-644b72d4c65a","Type":"ContainerStarted","Data":"687e1fda801e5728ffe3ec8d6744fd12d054177212ffb6b0f67c4d02a1e2470f"}
Nov 02 20:17:21.648033 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:21.647952    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"
Nov 02 20:17:21.766466 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:21.766405    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs" podStartSLOduration=1.766389577 podStartE2EDuration="1.766389577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:21.763236753 +0000 UTC m=+1722.760925359" watchObservedRunningTime="2026-11-02 20:17:21.766389577 +0000 UTC m=+1722.764078186"
Nov 02 20:17:24.105051 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:24.104996    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:17:25.029907 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:25.029869    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:26.226592 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:26.226554    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:27.360545 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:17:27.360508    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:17:31.236062 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:31.236024    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:31.465917 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:31.465875    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:32.656877 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.656839    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"
Nov 02 20:17:32.678279 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.678245    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"]
Nov 02 20:17:32.678875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.678855    5224 cpu_manager.go:414] "containerMap: removing container" podUID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a" containerName="cleanup"
Nov 02 20:17:32.678997 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.678876    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf16df4-48e0-417b-8a03-82cf8fb4a10a" containerName="cleanup"
Nov 02 20:17:32.678997 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.678891    5224 cpu_manager.go:414] "containerMap: removing container" podUID="95f31166-a27b-4a71-98d0-a727faea4589" containerName="cleanup"
Nov 02 20:17:32.678997 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.678899    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f31166-a27b-4a71-98d0-a727faea4589" containerName="cleanup"
Nov 02 20:17:32.678997 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.678955    5224 cpu_manager.go:414] "containerMap: removing container" podUID="2189a356-2417-4635-b1b4-22f3920a5f21" containerName="cleanup"
Nov 02 20:17:32.678997 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.678963    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2189a356-2417-4635-b1b4-22f3920a5f21" containerName="cleanup"
Nov 02 20:17:32.679251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.679117    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c3b61e-5330-4cab-8e14-235fefda4c8c" containerName="istio-proxy"
Nov 02 20:17:32.679251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.679131    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="2189a356-2417-4635-b1b4-22f3920a5f21" containerName="cleanup"
Nov 02 20:17:32.679251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.679142    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b298bc-073a-49e8-86ae-998b10102bfe" containerName="cleanup"
Nov 02 20:17:32.679251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.679153    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4614db6-6d7e-4ae9-bcc7-b1d746a45b8d" containerName="cleanup"
Nov 02 20:17:32.679251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.679175    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="63752026-3fce-43fd-8eb5-b0dfb4e4c398" containerName="cleanup"
Nov 02 20:17:32.682762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.682737    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.736326 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.736292    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5e04421b-openshift-default-dockercfg-b5655\""
Nov 02 20:17:32.767430 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.767395    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"]
Nov 02 20:17:32.795828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.795790    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.796107 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.796084    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.796291 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.796258    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.796408 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.796393    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.796538 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.796521    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.796676 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.796661    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.797380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.797339    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.797569 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.797545    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.797697 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.797658    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4t6f\" (UniqueName: \"kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-kube-api-access-n4t6f\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.898764 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.898717    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4t6f\" (UniqueName: \"kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-kube-api-access-n4t6f\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.899025 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.898913    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.899025 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.898961    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.899226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.899078    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.899226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.899215    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.899343 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.899255    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.899343 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.899294    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.899452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.899380    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.899452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.899412    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.900060 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.900024    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.900198 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.900138    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.900198 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.900145    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.900332 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.900309    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.900555 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.900535    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.901887 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.901827    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.906211 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.906190    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.956915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.956842    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.956915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.956843    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4t6f\" (UniqueName: \"kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-kube-api-access-n4t6f\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:32.997134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:32.997097    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:33.415340 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:17:33.415306    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3468f2_ef1e_4554_b6c7_e5e1d6ac4d4f.slice/crio-937ea30a5b627897af6989693a42a4d8df5939c071cbd49e4325499655251e17 WatchSource:0}: Error finding container 937ea30a5b627897af6989693a42a4d8df5939c071cbd49e4325499655251e17: Status 404 returned error can't find the container with id 937ea30a5b627897af6989693a42a4d8df5939c071cbd49e4325499655251e17
Nov 02 20:17:33.417908 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:33.417873    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:17:33.418027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:33.417943    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:17:33.418027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:33.417974    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:17:33.419014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:33.418992    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"]
Nov 02 20:17:33.722934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:33.722825    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws" event={"ID":"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f","Type":"ContainerStarted","Data":"e057f3815f88fc07967b44df65bceb07950d7228b0a308bc6d53bec47a3e7c09"}
Nov 02 20:17:33.722934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:33.722902    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws" event={"ID":"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f","Type":"ContainerStarted","Data":"937ea30a5b627897af6989693a42a4d8df5939c071cbd49e4325499655251e17"}
Nov 02 20:17:33.842020 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:33.841962    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws" podStartSLOduration=1.841946071 podStartE2EDuration="1.841946071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:33.841016324 +0000 UTC m=+1734.838704933" watchObservedRunningTime="2026-11-02 20:17:33.841946071 +0000 UTC m=+1734.839634674"
Nov 02 20:17:33.905450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:33.905413    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:33.997546 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:33.997449    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:35.002145 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:35.002103    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 20:17:35.002583 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:35.002192    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws" podUID="cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 20:17:35.998118 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:35.998074    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 20:17:35.998312 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:35.998153    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws" podUID="cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 20:17:36.091603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:36.091543    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"]
Nov 02 20:17:36.199537 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:36.199496    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:36.982705 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:36.982664    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:36.997830 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:36.997756    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 20:17:36.998072 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:36.997862    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws" podUID="cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 20:17:37.676637 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.676584    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"]
Nov 02 20:17:37.681465 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.681438    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.737358 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.737317    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-40aecd-openshift-default-dockercfg-sm8fz\""
Nov 02 20:17:37.789343 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.789287    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.789743 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.789719    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4xdh\" (UniqueName: \"kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-kube-api-access-g4xdh\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.789929 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.789904    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.790093 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.789952    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.790093 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.789984    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.790093 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.790089    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.790338 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.790194    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.790338 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.790220    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.790338 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.790274    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.795441 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.795414    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"]
Nov 02 20:17:37.892106 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.892060    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.892320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.892128    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.892320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.892180    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.892320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.892298    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.892501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.892391    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4xdh\" (UniqueName: \"kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-kube-api-access-g4xdh\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.892501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.892444    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.892501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.892486    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.892665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.892530    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.892722 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.892670    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.892864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.892842    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.893074 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.893056    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.893568 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.893549    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.893980 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.893954    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.894182 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.894157    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.895701 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.895663    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.895854 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.895836    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.977809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.976623    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4xdh\" (UniqueName: \"kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-kube-api-access-g4xdh\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.978217 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.978180    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.995877 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.995845    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:37.998046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.998021    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 20:17:37.998136 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:37.998070    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws" podUID="cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 20:17:38.475365 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:17:38.475325    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fb9c1f_b33a_4244_bd0b_ff2393f37587.slice/crio-78673ec2bcddb2fe275563dd4c159b19a6bb0d7a84fb6c9f1434936e87ec75da WatchSource:0}: Error finding container 78673ec2bcddb2fe275563dd4c159b19a6bb0d7a84fb6c9f1434936e87ec75da: Status 404 returned error can't find the container with id 78673ec2bcddb2fe275563dd4c159b19a6bb0d7a84fb6c9f1434936e87ec75da
Nov 02 20:17:38.476029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:38.475994    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"]
Nov 02 20:17:38.478520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:38.478477    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:17:38.478639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:38.478553    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:17:38.478639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:38.478583    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:17:38.783346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:38.783309    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd" event={"ID":"94fb9c1f-b33a-4244-bd0b-ff2393f37587","Type":"ContainerStarted","Data":"393a14fd3c4d3bc5cf3a0126f28f302c77f20f5999fe933865cc7b11c3288c26"}
Nov 02 20:17:38.783786 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:38.783367    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd" event={"ID":"94fb9c1f-b33a-4244-bd0b-ff2393f37587","Type":"ContainerStarted","Data":"78673ec2bcddb2fe275563dd4c159b19a6bb0d7a84fb6c9f1434936e87ec75da"}
Nov 02 20:17:38.900805 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:38.900735    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd" podStartSLOduration=1.900718249 podStartE2EDuration="1.900718249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:38.900057259 +0000 UTC m=+1739.897745870" watchObservedRunningTime="2026-11-02 20:17:38.900718249 +0000 UTC m=+1739.898406925"
Nov 02 20:17:38.996351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:38.996309    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:38.997611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:38.997588    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 20:17:38.997781 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:38.997632    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws" podUID="cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 20:17:40.005263 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:40.005210    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 20:17:40.005857 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:40.005284    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd" podUID="94fb9c1f-b33a-4244-bd0b-ff2393f37587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 20:17:40.206684 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:40.206584    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:40.207646 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:40.207612    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:40.207947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:40.207933    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:17:40.997061 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:40.997013    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 20:17:40.997256 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:40.997103    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd" podUID="94fb9c1f-b33a-4244-bd0b-ff2393f37587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 20:17:41.996467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:41.996425    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 20:17:41.996958 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:41.996515    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd" podUID="94fb9c1f-b33a-4244-bd0b-ff2393f37587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 20:17:42.084260 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:42.084210    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:42.997006 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:42.996969    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 20:17:42.997443 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:42.997035    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd" podUID="94fb9c1f-b33a-4244-bd0b-ff2393f37587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 20:17:43.997191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:43.997154    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 20:17:43.997660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:43.997212    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd" podUID="94fb9c1f-b33a-4244-bd0b-ff2393f37587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 20:17:45.005307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:45.005273    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:45.005738 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:45.005569    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:45.006284 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:45.006262    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:17:46.943178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:46.943130    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:47.119938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.119895    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:47.290971 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.290932    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:47.790201 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.790157    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"]
Nov 02 20:17:47.795824 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.795799    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.841986 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.841947    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"]
Nov 02 20:17:47.845305 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.845272    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.845463 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.845435    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.845556 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.845541    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.845620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.845575    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.845620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.845600    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qb77\" (UniqueName: \"kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-kube-api-access-5qb77\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.845723 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.845703    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.845795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.845744    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.845854 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.845793    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.845907 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.845894    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.846762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.846745    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:47.894147 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.894113    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"]
Nov 02 20:17:47.947085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947027    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:47.947618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947099    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:47.947618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947155    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.947618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947193    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.947618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947224    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qb77\" (UniqueName: \"kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-kube-api-access-5qb77\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.947618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947261    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kwtk\" (UniqueName: \"kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-kube-api-access-8kwtk\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:47.947618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947306    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.947618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947342    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:47.947618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947375    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.947618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947415    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.947618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947486    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:47.947618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947524    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.947618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947618    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.948295 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947699    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:47.948295 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947742    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:47.948295 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947798    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.948295 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947822    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:47.948295 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.947865    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:47.948789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.948746    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.949040 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.949012    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.949698 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.949677    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.949965 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.949933    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.950214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.950195    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.952858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.952836    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"]
Nov 02 20:17:47.953540 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.953514    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:47.953815 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:47.953747    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:48.008115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.008069    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qb77\" (UniqueName: \"kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-kube-api-access-5qb77\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:48.008364 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.008342    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:48.049501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.049397    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.049501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.049465    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.049501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.049501    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.050262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.049539    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.050262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.049598    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.050262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.049627    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.050262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.049681    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kwtk\" (UniqueName: \"kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-kube-api-access-8kwtk\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.050262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.049732    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.050262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.049832    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.050262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.050057    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.050262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.050129    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.050635 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.050317    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.050635 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.050328    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.050635 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.050600    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.052273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.052250    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.052405 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.052385    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.107625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.107585    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.107829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.107707    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kwtk\" (UniqueName: \"kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-kube-api-access-8kwtk\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.110439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.110413    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:48.176809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.175065    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:48.465341 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.465257    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:48.533968 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:17:48.533936    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14b2afc_92d5_4948_aa9a_fc1d271659fe.slice/crio-1fd89bdc4c8d1e1737a05bf65a4815d9f282121b8c6d4803bdfce2706178a05f WatchSource:0}: Error finding container 1fd89bdc4c8d1e1737a05bf65a4815d9f282121b8c6d4803bdfce2706178a05f: Status 404 returned error can't find the container with id 1fd89bdc4c8d1e1737a05bf65a4815d9f282121b8c6d4803bdfce2706178a05f
Nov 02 20:17:48.536671 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.536631    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:17:48.536814 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.536719    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:17:48.536814 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.536802    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:17:48.537662 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.537636    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"]
Nov 02 20:17:48.586873 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:17:48.586843    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6529d3d9_8daa_4f3d_b0bd_3bf157236d3e.slice/crio-2052afa4791dbbe9c37a89ddbd73a40ac78a8b3440e5f7edb35c06d7dab77307 WatchSource:0}: Error finding container 2052afa4791dbbe9c37a89ddbd73a40ac78a8b3440e5f7edb35c06d7dab77307: Status 404 returned error can't find the container with id 2052afa4791dbbe9c37a89ddbd73a40ac78a8b3440e5f7edb35c06d7dab77307
Nov 02 20:17:48.589871 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.589833    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:17:48.589996 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.589907    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:17:48.589996 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.589938    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:17:48.590273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.590248    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"]
Nov 02 20:17:48.860702 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.860657    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq" event={"ID":"a14b2afc-92d5-4948-aa9a-fc1d271659fe","Type":"ContainerStarted","Data":"868a4b34177321fd909bd4994a098d391aab5e12b46f9ec4b08a78c446671887"}
Nov 02 20:17:48.860967 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.860731    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq" event={"ID":"a14b2afc-92d5-4948-aa9a-fc1d271659fe","Type":"ContainerStarted","Data":"1fd89bdc4c8d1e1737a05bf65a4815d9f282121b8c6d4803bdfce2706178a05f"}
Nov 02 20:17:48.863057 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.863024    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m" event={"ID":"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e","Type":"ContainerStarted","Data":"b1e4b9d60766b16a62e7cbcd83f6dccbf512b2eeb2a28d0c290fec18ac560469"}
Nov 02 20:17:48.863227 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.863080    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m" event={"ID":"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e","Type":"ContainerStarted","Data":"2052afa4791dbbe9c37a89ddbd73a40ac78a8b3440e5f7edb35c06d7dab77307"}
Nov 02 20:17:48.980130 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:48.980058    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq" podStartSLOduration=1.980039272 podStartE2EDuration="1.980039272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:48.978152706 +0000 UTC m=+1749.975841317" watchObservedRunningTime="2026-11-02 20:17:48.980039272 +0000 UTC m=+1749.977727882"
Nov 02 20:17:49.098309 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:49.098246    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m" podStartSLOduration=2.09823043 podStartE2EDuration="2.09823043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:49.096995193 +0000 UTC m=+1750.094683803" watchObservedRunningTime="2026-11-02 20:17:49.09823043 +0000 UTC m=+1750.095919040"
Nov 02 20:17:49.110959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:49.110873    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:49.112476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:49.112449    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 20:17:49.112598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:49.112506    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq" podUID="a14b2afc-92d5-4948-aa9a-fc1d271659fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 20:17:49.175969 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:49.175927    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:49.178982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:49.178941    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:17:49.179149 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:49.178999    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m" podUID="6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 20:17:49.934231 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:49.934199    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:50.120206 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:50.112213    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 20:17:50.120206 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:50.112283    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq" podUID="a14b2afc-92d5-4948-aa9a-fc1d271659fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 20:17:50.184858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:50.180940    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:17:50.184858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:50.181006    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m" podUID="6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 20:17:51.113327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.113128    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 20:17:51.113327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.113099    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs" podUID="835fa46e-2a18-4009-a5a4-51e3599c161a" containerName="main" containerID="cri-o://1ea832f86a1043f3cb559c9113394e3feb9bad661bc372b1a5a589495830af09" gracePeriod=45
Nov 02 20:17:51.113327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.113191    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq" podUID="a14b2afc-92d5-4948-aa9a-fc1d271659fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 20:17:51.197870 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.196975    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:17:51.197870 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.197066    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m" podUID="6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 20:17:51.540897 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.540865    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:17:51.634958 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.634924    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-home\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-home\") pod \"835fa46e-2a18-4009-a5a4-51e3599c161a\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") "
Nov 02 20:17:51.635149 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.634968    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835fa46e-2a18-4009-a5a4-51e3599c161a-kube-api-access-8tdpb\" (UniqueName: \"kubernetes.io/projected/835fa46e-2a18-4009-a5a4-51e3599c161a-kube-api-access-8tdpb\") pod \"835fa46e-2a18-4009-a5a4-51e3599c161a\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") "
Nov 02 20:17:51.635149 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.635005    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-dshm\") pod \"835fa46e-2a18-4009-a5a4-51e3599c161a\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") "
Nov 02 20:17:51.635149 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.635104    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/835fa46e-2a18-4009-a5a4-51e3599c161a-tls-certs\" (UniqueName: \"kubernetes.io/secret/835fa46e-2a18-4009-a5a4-51e3599c161a-tls-certs\") pod \"835fa46e-2a18-4009-a5a4-51e3599c161a\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") "
Nov 02 20:17:51.635149 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.635124    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-tmp-dir\") pod \"835fa46e-2a18-4009-a5a4-51e3599c161a\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") "
Nov 02 20:17:51.635149 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.635149    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-model-cache\") pod \"835fa46e-2a18-4009-a5a4-51e3599c161a\" (UID: \"835fa46e-2a18-4009-a5a4-51e3599c161a\") "
Nov 02 20:17:51.635727 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.635688    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-tmp-dir" pod "835fa46e-2a18-4009-a5a4-51e3599c161a" (UID: "835fa46e-2a18-4009-a5a4-51e3599c161a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:51.635727 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.635714    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-model-cache" pod "835fa46e-2a18-4009-a5a4-51e3599c161a" (UID: "835fa46e-2a18-4009-a5a4-51e3599c161a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:51.635877 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.635836    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-home" pod "835fa46e-2a18-4009-a5a4-51e3599c161a" (UID: "835fa46e-2a18-4009-a5a4-51e3599c161a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:51.637500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.637481    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835fa46e-2a18-4009-a5a4-51e3599c161a-tls-certs" pod "835fa46e-2a18-4009-a5a4-51e3599c161a" (UID: "835fa46e-2a18-4009-a5a4-51e3599c161a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:17:51.637683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.637662    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835fa46e-2a18-4009-a5a4-51e3599c161a-kube-api-access-8tdpb" pod "835fa46e-2a18-4009-a5a4-51e3599c161a" (UID: "835fa46e-2a18-4009-a5a4-51e3599c161a"). InnerVolumeSpecName "kube-api-access-8tdpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:51.637683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.637672    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-dshm" pod "835fa46e-2a18-4009-a5a4-51e3599c161a" (UID: "835fa46e-2a18-4009-a5a4-51e3599c161a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:51.736884 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.736841    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:51.736884 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.736885    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:51.737119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.736900    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tdpb\" (UniqueName: \"kubernetes.io/projected/835fa46e-2a18-4009-a5a4-51e3599c161a-kube-api-access-8tdpb\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:51.737119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.736913    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:51.737119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.736926    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835fa46e-2a18-4009-a5a4-51e3599c161a-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:51.737119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.736968    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835fa46e-2a18-4009-a5a4-51e3599c161a-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:51.917673 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.917638    5224 generic.go:356] "Generic (PLEG): container finished" podID="835fa46e-2a18-4009-a5a4-51e3599c161a" containerID="1ea832f86a1043f3cb559c9113394e3feb9bad661bc372b1a5a589495830af09" exitCode=0
Nov 02 20:17:51.917673 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.917685    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs" event={"ID":"835fa46e-2a18-4009-a5a4-51e3599c161a","Type":"ContainerDied","Data":"1ea832f86a1043f3cb559c9113394e3feb9bad661bc372b1a5a589495830af09"}
Nov 02 20:17:51.917978 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.917692    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"
Nov 02 20:17:51.917978 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.917714    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs" event={"ID":"835fa46e-2a18-4009-a5a4-51e3599c161a","Type":"ContainerDied","Data":"672883f5b62a67f1109aa996ad3159593f5a2652c4d20bd9df60543a37f03d28"}
Nov 02 20:17:51.917978 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.917731    5224 scope.go:122] "RemoveContainer" containerID="1ea832f86a1043f3cb559c9113394e3feb9bad661bc372b1a5a589495830af09"
Nov 02 20:17:51.929380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.929353    5224 scope.go:122] "RemoveContainer" containerID="1ea832f86a1043f3cb559c9113394e3feb9bad661bc372b1a5a589495830af09"
Nov 02 20:17:51.929757 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:17:51.929736    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea832f86a1043f3cb559c9113394e3feb9bad661bc372b1a5a589495830af09\": container with ID starting with 1ea832f86a1043f3cb559c9113394e3feb9bad661bc372b1a5a589495830af09 not found: ID does not exist" containerID="1ea832f86a1043f3cb559c9113394e3feb9bad661bc372b1a5a589495830af09"
Nov 02 20:17:51.929828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:51.929785    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea832f86a1043f3cb559c9113394e3feb9bad661bc372b1a5a589495830af09"} err="failed to get container status \"1ea832f86a1043f3cb559c9113394e3feb9bad661bc372b1a5a589495830af09\": rpc error: code = NotFound desc = could not find container \"1ea832f86a1043f3cb559c9113394e3feb9bad661bc372b1a5a589495830af09\": container with ID starting with 1ea832f86a1043f3cb559c9113394e3feb9bad661bc372b1a5a589495830af09 not found: ID does not exist"
Nov 02 20:17:52.100795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:52.100358    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"]
Nov 02 20:17:52.111653 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:52.111612    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 20:17:52.111858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:52.111677    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq" podUID="a14b2afc-92d5-4948-aa9a-fc1d271659fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 20:17:52.145354 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:52.145313    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5e04421b/e2e-lifecycle-model-5e04421b-kserve-66f6c54697-vj2xs"]
Nov 02 20:17:52.176857 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:52.176035    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:17:52.176857 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:52.176113    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m" podUID="6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 20:17:52.417911 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:52.417875    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:53.111342 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:53.111299    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 20:17:53.111515 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:53.111357    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq" podUID="a14b2afc-92d5-4948-aa9a-fc1d271659fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 20:17:53.178793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:53.178071    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:17:53.178793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:53.178148    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m" podUID="6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 20:17:53.191804 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:53.190904    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835fa46e-2a18-4009-a5a4-51e3599c161a" path="/var/lib/kubelet/pods/835fa46e-2a18-4009-a5a4-51e3599c161a/volumes"
Nov 02 20:17:54.111411 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:54.111371    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 20:17:54.111855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:54.111438    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq" podUID="a14b2afc-92d5-4948-aa9a-fc1d271659fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 20:17:54.177794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:54.176960    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:17:54.177794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:54.177023    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m" podUID="6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 20:17:54.516188 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:54.516155    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:55.111691 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:55.111647    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 20:17:55.112165 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:55.111722    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq" podUID="a14b2afc-92d5-4948-aa9a-fc1d271659fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 20:17:55.175690 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:55.175653    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:17:55.175895 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:55.175718    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m" podUID="6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 20:17:55.342483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:55.342446    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:55.730783 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:55.730723    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:56.116320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:56.116292    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:56.116843 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:56.116607    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:56.117405 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:56.117381    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:17:56.180781 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:56.180735    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:56.181048 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:56.181029    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:56.181684 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:56.181662    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:17:56.410999 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:56.410907    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"]
Nov 02 20:17:56.817933 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:56.817897    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:57.036086 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:57.036048    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:57.361673 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:17:57.361634    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:17:57.792484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:57.792445    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:57.967740 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:57.967701    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:17:58.167601 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:17:58.167496    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:01.343205 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:01.343161    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:01.412026 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:01.411978    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5" podUID="77cae86a-4c75-452c-8ac5-7845f9fbb985" containerName="payload-processing" containerID="cri-o://2bfd774f43116e88df40fc2fd70415dd13a06ab4b83cbc0511f67bf0efe42789" gracePeriod=25
Nov 02 20:18:01.678190 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:01.678161    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"
Nov 02 20:18:01.739654 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:01.739617    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77cae86a-4c75-452c-8ac5-7845f9fbb985-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77cae86a-4c75-452c-8ac5-7845f9fbb985-plugins-config-volume\") pod \"77cae86a-4c75-452c-8ac5-7845f9fbb985\" (UID: \"77cae86a-4c75-452c-8ac5-7845f9fbb985\") "
Nov 02 20:18:01.739906 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:01.739674    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77cae86a-4c75-452c-8ac5-7845f9fbb985-kube-api-access-tmcf9\" (UniqueName: \"kubernetes.io/projected/77cae86a-4c75-452c-8ac5-7845f9fbb985-kube-api-access-tmcf9\") pod \"77cae86a-4c75-452c-8ac5-7845f9fbb985\" (UID: \"77cae86a-4c75-452c-8ac5-7845f9fbb985\") "
Nov 02 20:18:01.740053 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:01.740028    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cae86a-4c75-452c-8ac5-7845f9fbb985-plugins-config-volume" pod "77cae86a-4c75-452c-8ac5-7845f9fbb985" (UID: "77cae86a-4c75-452c-8ac5-7845f9fbb985"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:18:01.741877 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:01.741853    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cae86a-4c75-452c-8ac5-7845f9fbb985-kube-api-access-tmcf9" pod "77cae86a-4c75-452c-8ac5-7845f9fbb985" (UID: "77cae86a-4c75-452c-8ac5-7845f9fbb985"). InnerVolumeSpecName "kube-api-access-tmcf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:01.841709 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:01.841670    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77cae86a-4c75-452c-8ac5-7845f9fbb985-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:01.841709 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:01.841707    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmcf9\" (UniqueName: \"kubernetes.io/projected/77cae86a-4c75-452c-8ac5-7845f9fbb985-kube-api-access-tmcf9\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:02.008046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.007953    5224 generic.go:356] "Generic (PLEG): container finished" podID="77cae86a-4c75-452c-8ac5-7845f9fbb985" containerID="2bfd774f43116e88df40fc2fd70415dd13a06ab4b83cbc0511f67bf0efe42789" exitCode=0
Nov 02 20:18:02.008046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.008033    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5" event={"ID":"77cae86a-4c75-452c-8ac5-7845f9fbb985","Type":"ContainerDied","Data":"2bfd774f43116e88df40fc2fd70415dd13a06ab4b83cbc0511f67bf0efe42789"}
Nov 02 20:18:02.008313 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.008063    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"
Nov 02 20:18:02.008313 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.008087    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5" event={"ID":"77cae86a-4c75-452c-8ac5-7845f9fbb985","Type":"ContainerDied","Data":"6be8d36258d08c651c0affe57da9e307dea3d1e2e2ce3d428384a5dec47608e6"}
Nov 02 20:18:02.008313 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.008116    5224 scope.go:122] "RemoveContainer" containerID="2bfd774f43116e88df40fc2fd70415dd13a06ab4b83cbc0511f67bf0efe42789"
Nov 02 20:18:02.019575 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.019553    5224 scope.go:122] "RemoveContainer" containerID="2bfd774f43116e88df40fc2fd70415dd13a06ab4b83cbc0511f67bf0efe42789"
Nov 02 20:18:02.019910 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:18:02.019889    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfd774f43116e88df40fc2fd70415dd13a06ab4b83cbc0511f67bf0efe42789\": container with ID starting with 2bfd774f43116e88df40fc2fd70415dd13a06ab4b83cbc0511f67bf0efe42789 not found: ID does not exist" containerID="2bfd774f43116e88df40fc2fd70415dd13a06ab4b83cbc0511f67bf0efe42789"
Nov 02 20:18:02.019957 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.019920    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfd774f43116e88df40fc2fd70415dd13a06ab4b83cbc0511f67bf0efe42789"} err="failed to get container status \"2bfd774f43116e88df40fc2fd70415dd13a06ab4b83cbc0511f67bf0efe42789\": rpc error: code = NotFound desc = could not find container \"2bfd774f43116e88df40fc2fd70415dd13a06ab4b83cbc0511f67bf0efe42789\": container with ID starting with 2bfd774f43116e88df40fc2fd70415dd13a06ab4b83cbc0511f67bf0efe42789 not found: ID does not exist"
Nov 02 20:18:02.175616 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.175579    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"]
Nov 02 20:18:02.226693 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.226655    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-15203f-bff4c76f7-h7cr5"]
Nov 02 20:18:02.926677 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.926640    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"]
Nov 02 20:18:02.927219 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.927203    5224 cpu_manager.go:414] "containerMap: removing container" podUID="77cae86a-4c75-452c-8ac5-7845f9fbb985" containerName="payload-processing"
Nov 02 20:18:02.927290 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.927220    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77cae86a-4c75-452c-8ac5-7845f9fbb985" containerName="payload-processing"
Nov 02 20:18:02.927290 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.927242    5224 cpu_manager.go:414] "containerMap: removing container" podUID="835fa46e-2a18-4009-a5a4-51e3599c161a" containerName="main"
Nov 02 20:18:02.927290 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.927247    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835fa46e-2a18-4009-a5a4-51e3599c161a" containerName="main"
Nov 02 20:18:02.927432 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.927334    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="835fa46e-2a18-4009-a5a4-51e3599c161a" containerName="main"
Nov 02 20:18:02.927432 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.927348    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="77cae86a-4c75-452c-8ac5-7845f9fbb985" containerName="payload-processing"
Nov 02 20:18:02.932158 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:02.932136    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.040379 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.040341    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"]
Nov 02 20:18:03.056088 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.056056    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.056273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.056173    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.056273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.056238    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.056357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.056293    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.056357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.056329    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.056466 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.056450    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.056511 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.056484    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.056546 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.056514    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.056587 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.056577    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzf6c\" (UniqueName: \"kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-kube-api-access-wzf6c\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.157610 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.157558    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.157839 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.157691    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.157839 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.157737    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.157839 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.157808    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.158004 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.157861    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzf6c\" (UniqueName: \"kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-kube-api-access-wzf6c\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.158004 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.157976    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.158133 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.158005    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.158133 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.158070    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.158133 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.158085    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.158133 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.158106    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.158341 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.158185    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.158341 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.158199    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.158341 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.158328    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.159016 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.158990    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.160685 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.160651    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.160948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.160927    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.170472 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.170442    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77cae86a-4c75-452c-8ac5-7845f9fbb985" path="/var/lib/kubelet/pods/77cae86a-4c75-452c-8ac5-7845f9fbb985/volumes"
Nov 02 20:18:03.216084 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.216007    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.216084 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.216049    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzf6c\" (UniqueName: \"kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-kube-api-access-wzf6c\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.247573 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.247538    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:03.500077 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.499976    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:03.669933 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:18:03.669890    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2454a23_ca3d_43d3_9a6e_3ba05db8c99f.slice/crio-7bcef49ed70fa4ba8e846b36fb836c470cbd28081108ad160d82afd3aa5a88a7 WatchSource:0}: Error finding container 7bcef49ed70fa4ba8e846b36fb836c470cbd28081108ad160d82afd3aa5a88a7: Status 404 returned error can't find the container with id 7bcef49ed70fa4ba8e846b36fb836c470cbd28081108ad160d82afd3aa5a88a7
Nov 02 20:18:03.672898 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.672861    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:18:03.672999 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.672939    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:18:03.672999 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.672972    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:18:03.673642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:03.673619    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"]
Nov 02 20:18:04.034024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:04.033985    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9" event={"ID":"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f","Type":"ContainerStarted","Data":"7aa42382f8c3bfc87b4fd6a7193fd0b15e284a3c300216bb51271633279b8fed"}
Nov 02 20:18:04.034508 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:04.034046    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9" event={"ID":"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f","Type":"ContainerStarted","Data":"7bcef49ed70fa4ba8e846b36fb836c470cbd28081108ad160d82afd3aa5a88a7"}
Nov 02 20:18:04.153236 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:04.153163    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9" podStartSLOduration=2.153143313 podStartE2EDuration="2.153143313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:04.151106794 +0000 UTC m=+1765.148795404" watchObservedRunningTime="2026-11-02 20:18:04.153143313 +0000 UTC m=+1765.150831923"
Nov 02 20:18:04.248514 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:04.248474    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:04.267789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:04.266920    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:18:04.267789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:04.266992    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9" podUID="b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 20:18:05.248759 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:05.248722    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:18:05.248759 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:05.248827    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9" podUID="b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 20:18:06.248618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:06.248580    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:18:06.248881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:06.248640    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9" podUID="b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 20:18:07.268795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:07.248832    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:18:07.268795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:07.248897    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9" podUID="b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 20:18:08.047934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:08.047896    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"]
Nov 02 20:18:08.099051 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:08.099013    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d1ceaf-glh4c"]
Nov 02 20:18:08.252790 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:08.248828    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:18:08.268794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:08.266148    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9" podUID="b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 20:18:08.908206 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:08.908167    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:09.183573 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:09.183491    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f" path="/var/lib/kubelet/pods/4d08a244-6b6f-4fb7-8b9a-8894a6bc1f4f/volumes"
Nov 02 20:18:09.248749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:09.248712    5224 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:18:09.248749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:09.248816    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9" podUID="b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 20:18:10.252881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:10.252843    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:10.253278 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:10.253135    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:10.253973 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:10.253952    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:10.789456 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:10.789424    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:18:14.198729 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:14.198684    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:14.368877 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:14.368836    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:19.087289 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:19.087232    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:19.546853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:19.546802    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:21.006852 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:21.006807    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:23.015520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.015483    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"]
Nov 02 20:18:23.021065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.021042    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.066869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.066827    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"]
Nov 02 20:18:23.071081 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.071051    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.114431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.114385    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.114623 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.114592    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.114672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.114628    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.114748 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.114726    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.114860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.114832    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.114906 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.114885    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a90fadb8-0255-48af-8269-1ff834043c25-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.114952 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.114928    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.115072 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.115050    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.115145 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.115093    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62583a4e-72b8-407f-ac57-9baa1a1705c0-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.115221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.115201    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.115263 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.115233    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.115343 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.115320    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpcnv\" (UniqueName: \"kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-kube-api-access-gpcnv\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.115391 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.115376    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a90fadb8-0255-48af-8269-1ff834043c25-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.115475 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.115458    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdj28\" (UniqueName: \"kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-kube-api-access-kdj28\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.115516 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.115489    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.115614 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.115600    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.115660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.115645    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.115706 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.115677    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.118409 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.118388    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"]
Nov 02 20:18:23.170972 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.170937    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"]
Nov 02 20:18:23.218442 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218398    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.218672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218456    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.218672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218517    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.218672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218544    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.218672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218576    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a90fadb8-0255-48af-8269-1ff834043c25-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.218672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218620    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.219027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218679    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.219027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218710    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62583a4e-72b8-407f-ac57-9baa1a1705c0-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.219027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218753    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.219027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218804    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.219027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218862    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpcnv\" (UniqueName: \"kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-kube-api-access-gpcnv\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.219027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218906    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a90fadb8-0255-48af-8269-1ff834043c25-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.219027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218953    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdj28\" (UniqueName: \"kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-kube-api-access-kdj28\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.219027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.218977    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.219352 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.219097    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.219352 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.219144    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.219352 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.219193    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.219352 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.219230    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.220606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.220268    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.220606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.220343    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.220606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.220409    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.220606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.220552    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.221837 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.221755    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a90fadb8-0255-48af-8269-1ff834043c25-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.223355 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.221930    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.223355 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.222654    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.223486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.223354    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a90fadb8-0255-48af-8269-1ff834043c25-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.223486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.222966    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62583a4e-72b8-407f-ac57-9baa1a1705c0-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.223486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.223051    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.223486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.223177    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.223645 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.223539    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.224547 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.224522    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.224825 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.224801    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.280027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.279934    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.280181 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.280017    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdj28\" (UniqueName: \"kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-kube-api-access-kdj28\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.280181 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.280036    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpcnv\" (UniqueName: \"kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-kube-api-access-gpcnv\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.280181 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.280125    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.335777 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.335739    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:23.385683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.385642    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:23.759280 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:18:23.759241    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62583a4e_72b8_407f_ac57_9baa1a1705c0.slice/crio-2ee03eaf9b1dc96197b0af7ddc41d1ae130aebbef77e1f7cff5d8cf69685cfe6 WatchSource:0}: Error finding container 2ee03eaf9b1dc96197b0af7ddc41d1ae130aebbef77e1f7cff5d8cf69685cfe6: Status 404 returned error can't find the container with id 2ee03eaf9b1dc96197b0af7ddc41d1ae130aebbef77e1f7cff5d8cf69685cfe6
Nov 02 20:18:23.761939 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.761903    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:18:23.762040 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.761974    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:18:23.762040 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.762006    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:18:23.762891 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.762865    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"]
Nov 02 20:18:23.807683 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:18:23.807656    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90fadb8_0255_48af_8269_1ff834043c25.slice/crio-d51afef576437a85ce581d796acf1756a759ce233dfb1e9ac637da0a99d4bc70 WatchSource:0}: Error finding container d51afef576437a85ce581d796acf1756a759ce233dfb1e9ac637da0a99d4bc70: Status 404 returned error can't find the container with id d51afef576437a85ce581d796acf1756a759ce233dfb1e9ac637da0a99d4bc70
Nov 02 20:18:23.812946 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.812919    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"]
Nov 02 20:18:23.815276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.815235    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:18:23.815363 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.815325    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:18:23.815407 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:23.815374    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:18:24.231885 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:24.231838    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562" event={"ID":"a90fadb8-0255-48af-8269-1ff834043c25","Type":"ContainerStarted","Data":"7b8fe8b3eac0be79d9e5967c401435fb5469efd908f9f00fed8c192e6c86df80"}
Nov 02 20:18:24.232444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:24.231907    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562" event={"ID":"a90fadb8-0255-48af-8269-1ff834043c25","Type":"ContainerStarted","Data":"d51afef576437a85ce581d796acf1756a759ce233dfb1e9ac637da0a99d4bc70"}
Nov 02 20:18:24.234549 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:24.234511    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22" event={"ID":"62583a4e-72b8-407f-ac57-9baa1a1705c0","Type":"ContainerStarted","Data":"ae45c40d52e6c59e304e9b04b0a86d93bdf41e4b6c6dc3e4ac12593655315477"}
Nov 02 20:18:24.234720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:24.234575    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22" event={"ID":"62583a4e-72b8-407f-ac57-9baa1a1705c0","Type":"ContainerStarted","Data":"2ee03eaf9b1dc96197b0af7ddc41d1ae130aebbef77e1f7cff5d8cf69685cfe6"}
Nov 02 20:18:24.336994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:24.336955    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:24.342618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:24.341878    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:18:24.342618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:24.341940    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22" podUID="62583a4e-72b8-407f-ac57-9baa1a1705c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 20:18:24.352580 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:24.352510    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562" podStartSLOduration=2.352484686 podStartE2EDuration="2.352484686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:24.348378488 +0000 UTC m=+1785.346067098" watchObservedRunningTime="2026-11-02 20:18:24.352484686 +0000 UTC m=+1785.350173298"
Nov 02 20:18:24.389315 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:24.389056    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:24.389315 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:24.389201    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:18:24.389315 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:24.389245    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562" podUID="a90fadb8-0255-48af-8269-1ff834043c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 20:18:25.225458 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:25.225416    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:25.336622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:25.336577    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:18:25.337112 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:25.336635    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22" podUID="62583a4e-72b8-407f-ac57-9baa1a1705c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 20:18:25.400834 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:25.399454    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:18:25.400834 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:25.399515    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562" podUID="a90fadb8-0255-48af-8269-1ff834043c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 20:18:25.409356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:25.409321    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:25.730313 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:25.730266    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:26.338917 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:26.338866    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:18:26.339715 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:26.339657    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22" podUID="62583a4e-72b8-407f-ac57-9baa1a1705c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 20:18:26.389672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:26.389532    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:18:26.389672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:26.389606    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562" podUID="a90fadb8-0255-48af-8269-1ff834043c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 20:18:26.631566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:26.631449    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22" podStartSLOduration=4.631430394 podStartE2EDuration="4.631430394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:24.478136244 +0000 UTC m=+1785.475824854" watchObservedRunningTime="2026-11-02 20:18:26.631430394 +0000 UTC m=+1787.629119004"
Nov 02 20:18:26.633424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:26.633401    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"]
Nov 02 20:18:26.633646 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:26.633624    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95" podUID="54740835-67a3-418f-a090-18345e77f536" containerName="istio-proxy" containerID="cri-o://9e42caf44b03d8057bbd63fe4a3b997bd05b2ac17a4a8854a5695a10f9d93f69" gracePeriod=30
Nov 02 20:18:27.023867 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.015558    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:27.336515 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.336427    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:18:27.336669 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.336508    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22" podUID="62583a4e-72b8-407f-ac57-9baa1a1705c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 20:18:27.400788 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.397401    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:18:27.400788 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.397483    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562" podUID="a90fadb8-0255-48af-8269-1ff834043c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 20:18:27.408456 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:18:27.408421    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:18:27.519910 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.519859    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"]
Nov 02 20:18:27.520302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.520278    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"]
Nov 02 20:18:27.520837 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.520793    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m" podUID="6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" containerName="istio-proxy" containerID="cri-o://b1e4b9d60766b16a62e7cbcd83f6dccbf512b2eeb2a28d0c290fec18ac560469" gracePeriod=30
Nov 02 20:18:27.521613 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.521315    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq" podUID="a14b2afc-92d5-4948-aa9a-fc1d271659fe" containerName="istio-proxy" containerID="cri-o://868a4b34177321fd909bd4994a098d391aab5e12b46f9ec4b08a78c446671887" gracePeriod=30
Nov 02 20:18:27.529055 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.529031    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"]
Nov 02 20:18:27.529442 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.529410    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws" podUID="cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" containerName="istio-proxy" containerID="cri-o://e057f3815f88fc07967b44df65bceb07950d7228b0a308bc6d53bec47a3e7c09" gracePeriod=30
Nov 02 20:18:27.534918 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.534873    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:27.535345 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.535308    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"]
Nov 02 20:18:27.535711 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:27.535675    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9" podUID="b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" containerName="istio-proxy" containerID="cri-o://7aa42382f8c3bfc87b4fd6a7193fd0b15e284a3c300216bb51271633279b8fed" gracePeriod=30
Nov 02 20:18:28.003993 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:28.003928    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:28.201805 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:28.201446    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:28.336192 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:28.336156    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:18:28.336384 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:28.336222    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22" podUID="62583a4e-72b8-407f-ac57-9baa1a1705c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 20:18:28.390803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:28.390442    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:18:28.390803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:28.390550    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562" podUID="a90fadb8-0255-48af-8269-1ff834043c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 20:18:29.337961 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:29.337904    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:18:29.338342 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:29.337987    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22" podUID="62583a4e-72b8-407f-ac57-9baa1a1705c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 20:18:29.388793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:29.388210    5224 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:18:29.388793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:29.388287    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562" podUID="a90fadb8-0255-48af-8269-1ff834043c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 20:18:29.535520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:29.535479    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:29.720147 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:29.720021    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:30.380254 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:30.380210    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:30.381079 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:30.380678    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:30.381802 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:30.381747    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:18:30.390991 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:30.390964    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:30.391286 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:30.391251    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:30.392151 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:30.392132    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:18:31.916355 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.916283    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:18:31.956215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956172    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-credential-socket\") pod \"54740835-67a3-418f-a090-18345e77f536\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") "
Nov 02 20:18:31.956435 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956231    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-socket\") pod \"54740835-67a3-418f-a090-18345e77f536\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") "
Nov 02 20:18:31.956435 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956252    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54740835-67a3-418f-a090-18345e77f536-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54740835-67a3-418f-a090-18345e77f536-istiod-ca-cert\") pod \"54740835-67a3-418f-a090-18345e77f536\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") "
Nov 02 20:18:31.956435 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956335    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-kube-api-access-bfgff\" (UniqueName: \"kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-kube-api-access-bfgff\") pod \"54740835-67a3-418f-a090-18345e77f536\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") "
Nov 02 20:18:31.956435 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956380    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-istio-token\" (UniqueName: \"kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-istio-token\") pod \"54740835-67a3-418f-a090-18345e77f536\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") "
Nov 02 20:18:31.956435 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956428    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-certs\") pod \"54740835-67a3-418f-a090-18345e77f536\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") "
Nov 02 20:18:31.956714 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956486    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-envoy\") pod \"54740835-67a3-418f-a090-18345e77f536\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") "
Nov 02 20:18:31.956714 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956510    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-credential-socket" pod "54740835-67a3-418f-a090-18345e77f536" (UID: "54740835-67a3-418f-a090-18345e77f536"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:31.956714 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956535    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-data\") pod \"54740835-67a3-418f-a090-18345e77f536\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") "
Nov 02 20:18:31.956714 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956619    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54740835-67a3-418f-a090-18345e77f536-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54740835-67a3-418f-a090-18345e77f536-istio-podinfo\") pod \"54740835-67a3-418f-a090-18345e77f536\" (UID: \"54740835-67a3-418f-a090-18345e77f536\") "
Nov 02 20:18:31.956714 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956641    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-socket" pod "54740835-67a3-418f-a090-18345e77f536" (UID: "54740835-67a3-418f-a090-18345e77f536"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:31.957040 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956804    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-certs" pod "54740835-67a3-418f-a090-18345e77f536" (UID: "54740835-67a3-418f-a090-18345e77f536"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:31.957040 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956920    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54740835-67a3-418f-a090-18345e77f536-istiod-ca-cert" pod "54740835-67a3-418f-a090-18345e77f536" (UID: "54740835-67a3-418f-a090-18345e77f536"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:18:31.957040 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.956918    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-data" pod "54740835-67a3-418f-a090-18345e77f536" (UID: "54740835-67a3-418f-a090-18345e77f536"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:31.957349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.957315    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:31.957349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.957343    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:31.957484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.957359    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54740835-67a3-418f-a090-18345e77f536-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:31.957484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.957374    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:31.957484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.957388    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:31.958753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.958722    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-istio-token" pod "54740835-67a3-418f-a090-18345e77f536" (UID: "54740835-67a3-418f-a090-18345e77f536"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:31.958924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.958795    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54740835-67a3-418f-a090-18345e77f536-istio-podinfo" pod "54740835-67a3-418f-a090-18345e77f536" (UID: "54740835-67a3-418f-a090-18345e77f536"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:18:31.959125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.959045    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-envoy" pod "54740835-67a3-418f-a090-18345e77f536" (UID: "54740835-67a3-418f-a090-18345e77f536"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:31.959178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:31.959155    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-kube-api-access-bfgff" pod "54740835-67a3-418f-a090-18345e77f536" (UID: "54740835-67a3-418f-a090-18345e77f536"). InnerVolumeSpecName "kube-api-access-bfgff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:32.058913 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.058874    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfgff\" (UniqueName: \"kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-kube-api-access-bfgff\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.058913 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.058911    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54740835-67a3-418f-a090-18345e77f536-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.058913 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.058921    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54740835-67a3-418f-a090-18345e77f536-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.059180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.058937    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54740835-67a3-418f-a090-18345e77f536-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.344383 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.344348    5224 generic.go:356] "Generic (PLEG): container finished" podID="54740835-67a3-418f-a090-18345e77f536" containerID="9e42caf44b03d8057bbd63fe4a3b997bd05b2ac17a4a8854a5695a10f9d93f69" exitCode=0
Nov 02 20:18:32.344577 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.344409    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"
Nov 02 20:18:32.344577 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.344426    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95" event={"ID":"54740835-67a3-418f-a090-18345e77f536","Type":"ContainerDied","Data":"9e42caf44b03d8057bbd63fe4a3b997bd05b2ac17a4a8854a5695a10f9d93f69"}
Nov 02 20:18:32.344577 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.344486    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95" event={"ID":"54740835-67a3-418f-a090-18345e77f536","Type":"ContainerDied","Data":"c759155346778165ba610dddf1ca543349b2dd5fc4aede16c14646de3d7c438d"}
Nov 02 20:18:32.344577 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.344508    5224 scope.go:122] "RemoveContainer" containerID="9e42caf44b03d8057bbd63fe4a3b997bd05b2ac17a4a8854a5695a10f9d93f69"
Nov 02 20:18:32.366967 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.366940    5224 scope.go:122] "RemoveContainer" containerID="9e42caf44b03d8057bbd63fe4a3b997bd05b2ac17a4a8854a5695a10f9d93f69"
Nov 02 20:18:32.367327 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:18:32.367299    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e42caf44b03d8057bbd63fe4a3b997bd05b2ac17a4a8854a5695a10f9d93f69\": container with ID starting with 9e42caf44b03d8057bbd63fe4a3b997bd05b2ac17a4a8854a5695a10f9d93f69 not found: ID does not exist" containerID="9e42caf44b03d8057bbd63fe4a3b997bd05b2ac17a4a8854a5695a10f9d93f69"
Nov 02 20:18:32.367402 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.367342    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e42caf44b03d8057bbd63fe4a3b997bd05b2ac17a4a8854a5695a10f9d93f69"} err="failed to get container status \"9e42caf44b03d8057bbd63fe4a3b997bd05b2ac17a4a8854a5695a10f9d93f69\": rpc error: code = NotFound desc = could not find container \"9e42caf44b03d8057bbd63fe4a3b997bd05b2ac17a4a8854a5695a10f9d93f69\": container with ID starting with 9e42caf44b03d8057bbd63fe4a3b997bd05b2ac17a4a8854a5695a10f9d93f69 not found: ID does not exist"
Nov 02 20:18:32.514723 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.514686    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"]
Nov 02 20:18:32.518485 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.518450    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-24m95"]
Nov 02 20:18:32.581495 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.581452    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:32.772159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.772124    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:18:32.803215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.803178    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:32.874162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.872341    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-envoy\") pod \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") "
Nov 02 20:18:32.874162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.872519    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istiod-ca-cert\") pod \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") "
Nov 02 20:18:32.874162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.872580    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-token\" (UniqueName: \"kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-token\") pod \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") "
Nov 02 20:18:32.874162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.872643    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-data\") pod \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") "
Nov 02 20:18:32.874162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.872717    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-certs\") pod \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") "
Nov 02 20:18:32.874162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.872757    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-podinfo\") pod \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") "
Nov 02 20:18:32.874162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.872830    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-kube-api-access-5qb77\" (UniqueName: \"kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-kube-api-access-5qb77\") pod \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") "
Nov 02 20:18:32.874162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.872871    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-credential-socket\") pod \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") "
Nov 02 20:18:32.874162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.872940    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-socket\") pod \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\" (UID: \"a14b2afc-92d5-4948-aa9a-fc1d271659fe\") "
Nov 02 20:18:32.874162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.873107    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istiod-ca-cert" pod "a14b2afc-92d5-4948-aa9a-fc1d271659fe" (UID: "a14b2afc-92d5-4948-aa9a-fc1d271659fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:18:32.874162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.873389    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-socket" pod "a14b2afc-92d5-4948-aa9a-fc1d271659fe" (UID: "a14b2afc-92d5-4948-aa9a-fc1d271659fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.874162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.873736    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.874162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.873781    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.875064 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.874212    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-credential-socket" pod "a14b2afc-92d5-4948-aa9a-fc1d271659fe" (UID: "a14b2afc-92d5-4948-aa9a-fc1d271659fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.875064 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.874338    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-certs" pod "a14b2afc-92d5-4948-aa9a-fc1d271659fe" (UID: "a14b2afc-92d5-4948-aa9a-fc1d271659fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.875064 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.874430    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-data" pod "a14b2afc-92d5-4948-aa9a-fc1d271659fe" (UID: "a14b2afc-92d5-4948-aa9a-fc1d271659fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.878065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.878021    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-token" pod "a14b2afc-92d5-4948-aa9a-fc1d271659fe" (UID: "a14b2afc-92d5-4948-aa9a-fc1d271659fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:32.878741 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.878653    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:32.879287 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.879198    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-kube-api-access-5qb77" pod "a14b2afc-92d5-4948-aa9a-fc1d271659fe" (UID: "a14b2afc-92d5-4948-aa9a-fc1d271659fe"). InnerVolumeSpecName "kube-api-access-5qb77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:32.879287 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.879253    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-envoy" pod "a14b2afc-92d5-4948-aa9a-fc1d271659fe" (UID: "a14b2afc-92d5-4948-aa9a-fc1d271659fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.880063 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.880033    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-podinfo" pod "a14b2afc-92d5-4948-aa9a-fc1d271659fe" (UID: "a14b2afc-92d5-4948-aa9a-fc1d271659fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:18:32.902648 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.902617    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:18:32.916001 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.915973    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:18:32.974673 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.974644    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istiod-ca-cert\") pod \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") "
Nov 02 20:18:32.975146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.974700    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-kube-api-access-8kwtk\" (UniqueName: \"kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-kube-api-access-8kwtk\") pod \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") "
Nov 02 20:18:32.975146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.974732    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-kube-api-access-n4t6f\" (UniqueName: \"kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-kube-api-access-n4t6f\") pod \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") "
Nov 02 20:18:32.975146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.974753    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istiod-ca-cert\") pod \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") "
Nov 02 20:18:32.975146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.974805    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-data\") pod \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") "
Nov 02 20:18:32.975146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.974843    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istiod-ca-cert\") pod \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") "
Nov 02 20:18:32.975146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.974978    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-credential-socket\") pod \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") "
Nov 02 20:18:32.975146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975029    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-envoy\") pod \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") "
Nov 02 20:18:32.975146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975064    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-token\" (UniqueName: \"kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-token\") pod \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") "
Nov 02 20:18:32.975146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975105    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-socket\") pod \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") "
Nov 02 20:18:32.975146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975132    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-data" pod "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" (UID: "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975151    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istiod-ca-cert" pod "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" (UID: "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975185    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-envoy\") pod \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") "
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975224    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-podinfo\") pod \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") "
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975263    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-kube-api-access-wzf6c\" (UniqueName: \"kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-kube-api-access-wzf6c\") pod \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") "
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975279    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-credential-socket" pod "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" (UID: "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975287    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istiod-ca-cert" pod "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" (UID: "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975298    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-podinfo\") pod \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") "
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975334    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-certs\") pod \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") "
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975366    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-podinfo\") pod \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") "
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975392    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-envoy\") pod \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") "
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975444    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-certs\") pod \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") "
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975505    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-credential-socket\") pod \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") "
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975554    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-socket\") pod \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") "
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975618    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-data\") pod \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") "
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975654    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-data\") pod \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\" (UID: \"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f\") "
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975659    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istiod-ca-cert" pod "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" (UID: "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:18:32.975703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975700    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-certs\") pod \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") "
Nov 02 20:18:32.976639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975759    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-token\" (UniqueName: \"kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-token\") pod \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") "
Nov 02 20:18:32.976639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975853    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-socket\") pod \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") "
Nov 02 20:18:32.976639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975895    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-token\" (UniqueName: \"kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-token\") pod \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\" (UID: \"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e\") "
Nov 02 20:18:32.976639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975965    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-credential-socket\") pod \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\" (UID: \"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f\") "
Nov 02 20:18:32.976639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.975974    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-socket" pod "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" (UID: "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.976639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976145    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-credential-socket" pod "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" (UID: "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976726    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976751    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976784    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976801    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976816    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976833    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976847    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976862    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qb77\" (UniqueName: \"kubernetes.io/projected/a14b2afc-92d5-4948-aa9a-fc1d271659fe-kube-api-access-5qb77\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976878    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976893    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976908    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a14b2afc-92d5-4948-aa9a-fc1d271659fe-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976924    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976938    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.976954    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:32.977732 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.977227    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-socket" pod "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" (UID: "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.977732 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.977450    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-data" pod "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" (UID: "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.977732 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.977518    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-certs" pod "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" (UID: "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.977732 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.977648    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-credential-socket" pod "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" (UID: "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.977732 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.977690    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-certs" pod "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" (UID: "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.978008 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.977756    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-data" pod "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" (UID: "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.978008 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.977859    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-kube-api-access-8kwtk" pod "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" (UID: "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e"). InnerVolumeSpecName "kube-api-access-8kwtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:32.978008 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.977871    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-certs" pod "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" (UID: "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.978008 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.977921    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-socket" pod "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" (UID: "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.978008 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.977974    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-podinfo" pod "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" (UID: "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:18:32.978280 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.978037    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-token" pod "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" (UID: "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:32.978280 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.978160    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-kube-api-access-n4t6f" pod "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" (UID: "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f"). InnerVolumeSpecName "kube-api-access-n4t6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:32.978394 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.978301    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-envoy" pod "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" (UID: "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.979603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.979574    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-envoy" pod "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" (UID: "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:32.979779 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.979747    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-podinfo" pod "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" (UID: "cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:18:32.979962 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.979943    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-token" pod "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" (UID: "6529d3d9-8daa-4f3d-b0bd-3bf157236d3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:32.980015 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.979964    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-token" pod "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" (UID: "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:32.980015 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.979969    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-kube-api-access-wzf6c" pod "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" (UID: "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f"). InnerVolumeSpecName "kube-api-access-wzf6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:32.980335 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.980319    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-podinfo" pod "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" (UID: "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:18:32.981011 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:32.980997    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-envoy" pod "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" (UID: "b2454a23-ca3d-43d3-9a6e-3ba05db8c99f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:33.078258 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078224    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4t6f\" (UniqueName: \"kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-kube-api-access-n4t6f\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078258 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078255    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078258 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078265    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078258 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078274    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078258 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078283    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078258 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078293    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzf6c\" (UniqueName: \"kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-kube-api-access-wzf6c\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078258 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078302    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078310    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078319    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078328    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078336    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078345    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078352    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078361    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078369    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078378    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078387    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078395    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078404    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.078650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.078416    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kwtk\" (UniqueName: \"kubernetes.io/projected/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e-kube-api-access-8kwtk\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:33.170589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.170489    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54740835-67a3-418f-a090-18345e77f536" path="/var/lib/kubelet/pods/54740835-67a3-418f-a090-18345e77f536/volumes"
Nov 02 20:18:33.192968 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:18:33.192939    5224 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2454a23_ca3d_43d3_9a6e_3ba05db8c99f.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 20:18:33.351822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.351762    5224 generic.go:356] "Generic (PLEG): container finished" podID="cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" containerID="e057f3815f88fc07967b44df65bceb07950d7228b0a308bc6d53bec47a3e7c09" exitCode=0
Nov 02 20:18:33.351822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.351823    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"
Nov 02 20:18:33.352070 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.351868    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws" event={"ID":"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f","Type":"ContainerDied","Data":"e057f3815f88fc07967b44df65bceb07950d7228b0a308bc6d53bec47a3e7c09"}
Nov 02 20:18:33.352070 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.351922    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws" event={"ID":"cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f","Type":"ContainerDied","Data":"937ea30a5b627897af6989693a42a4d8df5939c071cbd49e4325499655251e17"}
Nov 02 20:18:33.352070 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.351946    5224 scope.go:122] "RemoveContainer" containerID="e057f3815f88fc07967b44df65bceb07950d7228b0a308bc6d53bec47a3e7c09"
Nov 02 20:18:33.353414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.353390    5224 generic.go:356] "Generic (PLEG): container finished" podID="a14b2afc-92d5-4948-aa9a-fc1d271659fe" containerID="868a4b34177321fd909bd4994a098d391aab5e12b46f9ec4b08a78c446671887" exitCode=0
Nov 02 20:18:33.353520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.353503    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"
Nov 02 20:18:33.353520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.353504    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq" event={"ID":"a14b2afc-92d5-4948-aa9a-fc1d271659fe","Type":"ContainerDied","Data":"868a4b34177321fd909bd4994a098d391aab5e12b46f9ec4b08a78c446671887"}
Nov 02 20:18:33.353645 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.353556    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq" event={"ID":"a14b2afc-92d5-4948-aa9a-fc1d271659fe","Type":"ContainerDied","Data":"1fd89bdc4c8d1e1737a05bf65a4815d9f282121b8c6d4803bdfce2706178a05f"}
Nov 02 20:18:33.355153 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.355129    5224 generic.go:356] "Generic (PLEG): container finished" podID="b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" containerID="7aa42382f8c3bfc87b4fd6a7193fd0b15e284a3c300216bb51271633279b8fed" exitCode=0
Nov 02 20:18:33.355273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.355204    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"
Nov 02 20:18:33.355273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.355211    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9" event={"ID":"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f","Type":"ContainerDied","Data":"7aa42382f8c3bfc87b4fd6a7193fd0b15e284a3c300216bb51271633279b8fed"}
Nov 02 20:18:33.355273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.355256    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9" event={"ID":"b2454a23-ca3d-43d3-9a6e-3ba05db8c99f","Type":"ContainerDied","Data":"7bcef49ed70fa4ba8e846b36fb836c470cbd28081108ad160d82afd3aa5a88a7"}
Nov 02 20:18:33.358101 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.358079    5224 generic.go:356] "Generic (PLEG): container finished" podID="6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" containerID="b1e4b9d60766b16a62e7cbcd83f6dccbf512b2eeb2a28d0c290fec18ac560469" exitCode=0
Nov 02 20:18:33.358204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.358140    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"
Nov 02 20:18:33.358204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.358166    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m" event={"ID":"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e","Type":"ContainerDied","Data":"b1e4b9d60766b16a62e7cbcd83f6dccbf512b2eeb2a28d0c290fec18ac560469"}
Nov 02 20:18:33.358325 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.358204    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m" event={"ID":"6529d3d9-8daa-4f3d-b0bd-3bf157236d3e","Type":"ContainerDied","Data":"2052afa4791dbbe9c37a89ddbd73a40ac78a8b3440e5f7edb35c06d7dab77307"}
Nov 02 20:18:33.364514 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.364474    5224 scope.go:122] "RemoveContainer" containerID="e057f3815f88fc07967b44df65bceb07950d7228b0a308bc6d53bec47a3e7c09"
Nov 02 20:18:33.364807 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:18:33.364788    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e057f3815f88fc07967b44df65bceb07950d7228b0a308bc6d53bec47a3e7c09\": container with ID starting with e057f3815f88fc07967b44df65bceb07950d7228b0a308bc6d53bec47a3e7c09 not found: ID does not exist" containerID="e057f3815f88fc07967b44df65bceb07950d7228b0a308bc6d53bec47a3e7c09"
Nov 02 20:18:33.364877 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.364814    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e057f3815f88fc07967b44df65bceb07950d7228b0a308bc6d53bec47a3e7c09"} err="failed to get container status \"e057f3815f88fc07967b44df65bceb07950d7228b0a308bc6d53bec47a3e7c09\": rpc error: code = NotFound desc = could not find container \"e057f3815f88fc07967b44df65bceb07950d7228b0a308bc6d53bec47a3e7c09\": container with ID starting with e057f3815f88fc07967b44df65bceb07950d7228b0a308bc6d53bec47a3e7c09 not found: ID does not exist"
Nov 02 20:18:33.364877 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.364833    5224 scope.go:122] "RemoveContainer" containerID="868a4b34177321fd909bd4994a098d391aab5e12b46f9ec4b08a78c446671887"
Nov 02 20:18:33.374642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.374620    5224 scope.go:122] "RemoveContainer" containerID="868a4b34177321fd909bd4994a098d391aab5e12b46f9ec4b08a78c446671887"
Nov 02 20:18:33.374956 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:18:33.374934    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868a4b34177321fd909bd4994a098d391aab5e12b46f9ec4b08a78c446671887\": container with ID starting with 868a4b34177321fd909bd4994a098d391aab5e12b46f9ec4b08a78c446671887 not found: ID does not exist" containerID="868a4b34177321fd909bd4994a098d391aab5e12b46f9ec4b08a78c446671887"
Nov 02 20:18:33.375033 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.374969    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868a4b34177321fd909bd4994a098d391aab5e12b46f9ec4b08a78c446671887"} err="failed to get container status \"868a4b34177321fd909bd4994a098d391aab5e12b46f9ec4b08a78c446671887\": rpc error: code = NotFound desc = could not find container \"868a4b34177321fd909bd4994a098d391aab5e12b46f9ec4b08a78c446671887\": container with ID starting with 868a4b34177321fd909bd4994a098d391aab5e12b46f9ec4b08a78c446671887 not found: ID does not exist"
Nov 02 20:18:33.375033 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.374996    5224 scope.go:122] "RemoveContainer" containerID="7aa42382f8c3bfc87b4fd6a7193fd0b15e284a3c300216bb51271633279b8fed"
Nov 02 20:18:33.386603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.386574    5224 scope.go:122] "RemoveContainer" containerID="7aa42382f8c3bfc87b4fd6a7193fd0b15e284a3c300216bb51271633279b8fed"
Nov 02 20:18:33.386971 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:18:33.386949    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa42382f8c3bfc87b4fd6a7193fd0b15e284a3c300216bb51271633279b8fed\": container with ID starting with 7aa42382f8c3bfc87b4fd6a7193fd0b15e284a3c300216bb51271633279b8fed not found: ID does not exist" containerID="7aa42382f8c3bfc87b4fd6a7193fd0b15e284a3c300216bb51271633279b8fed"
Nov 02 20:18:33.387029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.386984    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa42382f8c3bfc87b4fd6a7193fd0b15e284a3c300216bb51271633279b8fed"} err="failed to get container status \"7aa42382f8c3bfc87b4fd6a7193fd0b15e284a3c300216bb51271633279b8fed\": rpc error: code = NotFound desc = could not find container \"7aa42382f8c3bfc87b4fd6a7193fd0b15e284a3c300216bb51271633279b8fed\": container with ID starting with 7aa42382f8c3bfc87b4fd6a7193fd0b15e284a3c300216bb51271633279b8fed not found: ID does not exist"
Nov 02 20:18:33.387029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.387007    5224 scope.go:122] "RemoveContainer" containerID="b1e4b9d60766b16a62e7cbcd83f6dccbf512b2eeb2a28d0c290fec18ac560469"
Nov 02 20:18:33.398086 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.398062    5224 scope.go:122] "RemoveContainer" containerID="b1e4b9d60766b16a62e7cbcd83f6dccbf512b2eeb2a28d0c290fec18ac560469"
Nov 02 20:18:33.398417 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:18:33.398395    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e4b9d60766b16a62e7cbcd83f6dccbf512b2eeb2a28d0c290fec18ac560469\": container with ID starting with b1e4b9d60766b16a62e7cbcd83f6dccbf512b2eeb2a28d0c290fec18ac560469 not found: ID does not exist" containerID="b1e4b9d60766b16a62e7cbcd83f6dccbf512b2eeb2a28d0c290fec18ac560469"
Nov 02 20:18:33.398460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.398427    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e4b9d60766b16a62e7cbcd83f6dccbf512b2eeb2a28d0c290fec18ac560469"} err="failed to get container status \"b1e4b9d60766b16a62e7cbcd83f6dccbf512b2eeb2a28d0c290fec18ac560469\": rpc error: code = NotFound desc = could not find container \"b1e4b9d60766b16a62e7cbcd83f6dccbf512b2eeb2a28d0c290fec18ac560469\": container with ID starting with b1e4b9d60766b16a62e7cbcd83f6dccbf512b2eeb2a28d0c290fec18ac560469 not found: ID does not exist"
Nov 02 20:18:33.520667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.520634    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"]
Nov 02 20:18:33.523462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.523435    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-g778m"]
Nov 02 20:18:33.599596 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.599555    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:33.635813 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.635776    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"]
Nov 02 20:18:33.641157 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.641127    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-wwtm9"]
Nov 02 20:18:33.751840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.751799    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"]
Nov 02 20:18:33.755092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.755068    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-hjbws"]
Nov 02 20:18:33.865724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.865635    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"]
Nov 02 20:18:33.869207 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:33.869176    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-xwgtq"]
Nov 02 20:18:34.144568 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:34.144473    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:34.605150 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:34.605108    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"]
Nov 02 20:18:34.610222 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:34.605852    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6" podUID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerName="istio-proxy" containerID="cri-o://de9f36f0013b8d68b5b457b3fc9c5c10765b99f884513c4751396d34d7550def" gracePeriod=30
Nov 02 20:18:34.652348 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:34.652307    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:35.168470 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:35.168430    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" path="/var/lib/kubelet/pods/6529d3d9-8daa-4f3d-b0bd-3bf157236d3e/volumes"
Nov 02 20:18:35.169173 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:35.169150    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a14b2afc-92d5-4948-aa9a-fc1d271659fe" path="/var/lib/kubelet/pods/a14b2afc-92d5-4948-aa9a-fc1d271659fe/volumes"
Nov 02 20:18:35.169651 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:35.169632    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" path="/var/lib/kubelet/pods/b2454a23-ca3d-43d3-9a6e-3ba05db8c99f/volumes"
Nov 02 20:18:35.170466 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:35.170443    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" path="/var/lib/kubelet/pods/cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f/volumes"
Nov 02 20:18:35.424626 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:35.424537    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:37.805178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:37.805145    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:18:39.202897 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.201975    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:39.918890 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.918858    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:18:39.992406 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.992371    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-kube-api-access-h6vsr\" (UniqueName: \"kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-kube-api-access-h6vsr\") pod \"df701f2f-911b-40e0-9e88-a66aa003ee45\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") "
Nov 02 20:18:39.992598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.992459    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-envoy\") pod \"df701f2f-911b-40e0-9e88-a66aa003ee45\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") "
Nov 02 20:18:39.992598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.992509    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-podinfo\") pod \"df701f2f-911b-40e0-9e88-a66aa003ee45\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") "
Nov 02 20:18:39.992598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.992547    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-token\" (UniqueName: \"kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-token\") pod \"df701f2f-911b-40e0-9e88-a66aa003ee45\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") "
Nov 02 20:18:39.992598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.992587    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-socket\") pod \"df701f2f-911b-40e0-9e88-a66aa003ee45\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") "
Nov 02 20:18:39.992850 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.992608    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-certs\") pod \"df701f2f-911b-40e0-9e88-a66aa003ee45\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") "
Nov 02 20:18:39.992850 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.992719    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-data\") pod \"df701f2f-911b-40e0-9e88-a66aa003ee45\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") "
Nov 02 20:18:39.992958 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.992857    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-credential-socket\") pod \"df701f2f-911b-40e0-9e88-a66aa003ee45\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") "
Nov 02 20:18:39.992958 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.992903    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df701f2f-911b-40e0-9e88-a66aa003ee45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df701f2f-911b-40e0-9e88-a66aa003ee45-istiod-ca-cert\") pod \"df701f2f-911b-40e0-9e88-a66aa003ee45\" (UID: \"df701f2f-911b-40e0-9e88-a66aa003ee45\") "
Nov 02 20:18:39.993138 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.993059    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-certs" pod "df701f2f-911b-40e0-9e88-a66aa003ee45" (UID: "df701f2f-911b-40e0-9e88-a66aa003ee45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:39.993303 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.993257    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-socket" pod "df701f2f-911b-40e0-9e88-a66aa003ee45" (UID: "df701f2f-911b-40e0-9e88-a66aa003ee45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:39.993391 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.993348    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df701f2f-911b-40e0-9e88-a66aa003ee45-istiod-ca-cert" pod "df701f2f-911b-40e0-9e88-a66aa003ee45" (UID: "df701f2f-911b-40e0-9e88-a66aa003ee45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:18:39.993587 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.993567    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-credential-socket" pod "df701f2f-911b-40e0-9e88-a66aa003ee45" (UID: "df701f2f-911b-40e0-9e88-a66aa003ee45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:39.993683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.993647    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-data" pod "df701f2f-911b-40e0-9e88-a66aa003ee45" (UID: "df701f2f-911b-40e0-9e88-a66aa003ee45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:39.993802 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.993786    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:39.993873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.993804    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:39.993873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.993813    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df701f2f-911b-40e0-9e88-a66aa003ee45-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:39.993873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.993822    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:39.993873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.993833    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:39.995344 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.995257    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-token" pod "df701f2f-911b-40e0-9e88-a66aa003ee45" (UID: "df701f2f-911b-40e0-9e88-a66aa003ee45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:39.995344 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.995308    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-kube-api-access-h6vsr" pod "df701f2f-911b-40e0-9e88-a66aa003ee45" (UID: "df701f2f-911b-40e0-9e88-a66aa003ee45"). InnerVolumeSpecName "kube-api-access-h6vsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:39.995344 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.995333    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-envoy" pod "df701f2f-911b-40e0-9e88-a66aa003ee45" (UID: "df701f2f-911b-40e0-9e88-a66aa003ee45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:39.995530 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:39.995508    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-podinfo" pod "df701f2f-911b-40e0-9e88-a66aa003ee45" (UID: "df701f2f-911b-40e0-9e88-a66aa003ee45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:18:40.095342 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.095306    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6vsr\" (UniqueName: \"kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-kube-api-access-h6vsr\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:40.095342 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.095340    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:40.095549 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.095355    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:40.095549 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.095368    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df701f2f-911b-40e0-9e88-a66aa003ee45-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:40.412366 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.412328    5224 generic.go:356] "Generic (PLEG): container finished" podID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerID="de9f36f0013b8d68b5b457b3fc9c5c10765b99f884513c4751396d34d7550def" exitCode=0
Nov 02 20:18:40.412926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.412397    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"
Nov 02 20:18:40.412926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.412397    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6" event={"ID":"df701f2f-911b-40e0-9e88-a66aa003ee45","Type":"ContainerDied","Data":"de9f36f0013b8d68b5b457b3fc9c5c10765b99f884513c4751396d34d7550def"}
Nov 02 20:18:40.412926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.412456    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6" event={"ID":"df701f2f-911b-40e0-9e88-a66aa003ee45","Type":"ContainerDied","Data":"f972f31543d3a42adac5d4728d4231b5be247ab2ab00d93668ba6e41944658ac"}
Nov 02 20:18:40.412926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.412481    5224 scope.go:122] "RemoveContainer" containerID="de9f36f0013b8d68b5b457b3fc9c5c10765b99f884513c4751396d34d7550def"
Nov 02 20:18:40.423250 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.423231    5224 scope.go:122] "RemoveContainer" containerID="1c1f67f6f0d9510a4ada06d708f1c9b49b17fe13282f8bb1355d9419a8a0e696"
Nov 02 20:18:40.433052 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.433029    5224 scope.go:122] "RemoveContainer" containerID="de9f36f0013b8d68b5b457b3fc9c5c10765b99f884513c4751396d34d7550def"
Nov 02 20:18:40.433330 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:18:40.433310    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9f36f0013b8d68b5b457b3fc9c5c10765b99f884513c4751396d34d7550def\": container with ID starting with de9f36f0013b8d68b5b457b3fc9c5c10765b99f884513c4751396d34d7550def not found: ID does not exist" containerID="de9f36f0013b8d68b5b457b3fc9c5c10765b99f884513c4751396d34d7550def"
Nov 02 20:18:40.433373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.433342    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9f36f0013b8d68b5b457b3fc9c5c10765b99f884513c4751396d34d7550def"} err="failed to get container status \"de9f36f0013b8d68b5b457b3fc9c5c10765b99f884513c4751396d34d7550def\": rpc error: code = NotFound desc = could not find container \"de9f36f0013b8d68b5b457b3fc9c5c10765b99f884513c4751396d34d7550def\": container with ID starting with de9f36f0013b8d68b5b457b3fc9c5c10765b99f884513c4751396d34d7550def not found: ID does not exist"
Nov 02 20:18:40.433373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.433363    5224 scope.go:122] "RemoveContainer" containerID="1c1f67f6f0d9510a4ada06d708f1c9b49b17fe13282f8bb1355d9419a8a0e696"
Nov 02 20:18:40.433623 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:18:40.433606    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1f67f6f0d9510a4ada06d708f1c9b49b17fe13282f8bb1355d9419a8a0e696\": container with ID starting with 1c1f67f6f0d9510a4ada06d708f1c9b49b17fe13282f8bb1355d9419a8a0e696 not found: ID does not exist" containerID="1c1f67f6f0d9510a4ada06d708f1c9b49b17fe13282f8bb1355d9419a8a0e696"
Nov 02 20:18:40.433669 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.433629    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1f67f6f0d9510a4ada06d708f1c9b49b17fe13282f8bb1355d9419a8a0e696"} err="failed to get container status \"1c1f67f6f0d9510a4ada06d708f1c9b49b17fe13282f8bb1355d9419a8a0e696\": rpc error: code = NotFound desc = could not find container \"1c1f67f6f0d9510a4ada06d708f1c9b49b17fe13282f8bb1355d9419a8a0e696\": container with ID starting with 1c1f67f6f0d9510a4ada06d708f1c9b49b17fe13282f8bb1355d9419a8a0e696 not found: ID does not exist"
Nov 02 20:18:40.586160 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.586124    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"]
Nov 02 20:18:40.637800 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:40.637751    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-15203f-openshift-default-5b9ddbd8f5-9l4f6"]
Nov 02 20:18:41.172490 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:41.172455    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df701f2f-911b-40e0-9e88-a66aa003ee45" path="/var/lib/kubelet/pods/df701f2f-911b-40e0-9e88-a66aa003ee45/volumes"
Nov 02 20:18:41.305159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:41.305118    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:42.590688 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:42.589065    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:42.754243 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:42.754208    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:44.960190 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:44.960133    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:46.881645 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:46.881588    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:47.361748 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:47.361709    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:50.241308 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:50.241191    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:53.270279 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:53.270242    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:54.847591 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:54.847556    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:55.286697 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:55.286645    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:55.506075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:55.506033    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:55.682324 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:55.682198    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:55.858166 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:18:55.858123    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:18:57.360490 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:18:57.360447    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:19:00.859071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:00.859019    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:06.090579 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:06.090521    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:06.552732 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:06.552686    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:07.401101 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:07.401062    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:08.163642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:08.163584    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:08.353385 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:08.353345    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:08.394370 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:08.394330    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"]
Nov 02 20:19:08.394623 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:08.394590    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf" podUID="402ae8c8-446f-4812-9ecc-9cb29b9c35fe" containerName="istio-proxy" containerID="cri-o://fa0d25659638c0f643091b7f434e6d0d9e58219f659edff0c64e120bbd223913" gracePeriod=30
Nov 02 20:19:08.538325 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:08.538278    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:08.715761 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:08.715727    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:11.153121 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:11.153085    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:12.290923 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:12.290886    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:19:12.291418 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:12.290946    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf" podUID="402ae8c8-446f-4812-9ecc-9cb29b9c35fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:19:13.455756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.455716    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:13.673067 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.672994    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:19:13.673882 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.673854    5224 generic.go:356] "Generic (PLEG): container finished" podID="402ae8c8-446f-4812-9ecc-9cb29b9c35fe" containerID="fa0d25659638c0f643091b7f434e6d0d9e58219f659edff0c64e120bbd223913" exitCode=0
Nov 02 20:19:13.674022 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.673924    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf" event={"ID":"402ae8c8-446f-4812-9ecc-9cb29b9c35fe","Type":"ContainerDied","Data":"fa0d25659638c0f643091b7f434e6d0d9e58219f659edff0c64e120bbd223913"}
Nov 02 20:19:13.674022 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.673983    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf" event={"ID":"402ae8c8-446f-4812-9ecc-9cb29b9c35fe","Type":"ContainerDied","Data":"c0ace55e88124363ceb094e55819899c02af8611f07ec3e3b50c1cdf22d6c99f"}
Nov 02 20:19:13.674022 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.674008    5224 scope.go:122] "RemoveContainer" containerID="fa0d25659638c0f643091b7f434e6d0d9e58219f659edff0c64e120bbd223913"
Nov 02 20:19:13.681732 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.681702    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:13.687907 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.687848    5224 scope.go:122] "RemoveContainer" containerID="fa0d25659638c0f643091b7f434e6d0d9e58219f659edff0c64e120bbd223913"
Nov 02 20:19:13.688372 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:19:13.688340    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0d25659638c0f643091b7f434e6d0d9e58219f659edff0c64e120bbd223913\": container with ID starting with fa0d25659638c0f643091b7f434e6d0d9e58219f659edff0c64e120bbd223913 not found: ID does not exist" containerID="fa0d25659638c0f643091b7f434e6d0d9e58219f659edff0c64e120bbd223913"
Nov 02 20:19:13.688470 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.688379    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0d25659638c0f643091b7f434e6d0d9e58219f659edff0c64e120bbd223913"} err="failed to get container status \"fa0d25659638c0f643091b7f434e6d0d9e58219f659edff0c64e120bbd223913\": rpc error: code = NotFound desc = could not find container \"fa0d25659638c0f643091b7f434e6d0d9e58219f659edff0c64e120bbd223913\": container with ID starting with fa0d25659638c0f643091b7f434e6d0d9e58219f659edff0c64e120bbd223913 not found: ID does not exist"
Nov 02 20:19:13.695456 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.695428    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-certs\") pod \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") "
Nov 02 20:19:13.695549 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.695473    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-kube-api-access-mbmhc\" (UniqueName: \"kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-kube-api-access-mbmhc\") pod \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") "
Nov 02 20:19:13.695549 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.695507    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-data\") pod \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") "
Nov 02 20:19:13.695549 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.695530    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-token\" (UniqueName: \"kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-token\") pod \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") "
Nov 02 20:19:13.695705 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.695693    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-socket\") pod \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") "
Nov 02 20:19:13.696021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.695785    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-podinfo\") pod \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") "
Nov 02 20:19:13.696021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.695826    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istiod-ca-cert\") pod \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") "
Nov 02 20:19:13.696021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.695847    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-certs" pod "402ae8c8-446f-4812-9ecc-9cb29b9c35fe" (UID: "402ae8c8-446f-4812-9ecc-9cb29b9c35fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:13.696021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.695880    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-envoy\") pod \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") "
Nov 02 20:19:13.696021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.695932    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-credential-socket\") pod \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\" (UID: \"402ae8c8-446f-4812-9ecc-9cb29b9c35fe\") "
Nov 02 20:19:13.696021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.695877    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-data" pod "402ae8c8-446f-4812-9ecc-9cb29b9c35fe" (UID: "402ae8c8-446f-4812-9ecc-9cb29b9c35fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:13.696021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.695971    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-socket" pod "402ae8c8-446f-4812-9ecc-9cb29b9c35fe" (UID: "402ae8c8-446f-4812-9ecc-9cb29b9c35fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:13.696450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.696219    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-credential-socket" pod "402ae8c8-446f-4812-9ecc-9cb29b9c35fe" (UID: "402ae8c8-446f-4812-9ecc-9cb29b9c35fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:13.696450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.696266    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istiod-ca-cert" pod "402ae8c8-446f-4812-9ecc-9cb29b9c35fe" (UID: "402ae8c8-446f-4812-9ecc-9cb29b9c35fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:19:13.696558 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.696539    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:13.696632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.696561    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:13.696632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.696576    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:13.696632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.696592    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:13.696632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.696607    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:13.698083 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.698059    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-podinfo" pod "402ae8c8-446f-4812-9ecc-9cb29b9c35fe" (UID: "402ae8c8-446f-4812-9ecc-9cb29b9c35fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:19:13.698193 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.698101    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-envoy" pod "402ae8c8-446f-4812-9ecc-9cb29b9c35fe" (UID: "402ae8c8-446f-4812-9ecc-9cb29b9c35fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:13.698193 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.698118    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-token" pod "402ae8c8-446f-4812-9ecc-9cb29b9c35fe" (UID: "402ae8c8-446f-4812-9ecc-9cb29b9c35fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:13.698399 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.698379    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-kube-api-access-mbmhc" pod "402ae8c8-446f-4812-9ecc-9cb29b9c35fe" (UID: "402ae8c8-446f-4812-9ecc-9cb29b9c35fe"). InnerVolumeSpecName "kube-api-access-mbmhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:13.797575 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.797535    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:13.797575 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.797571    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:13.797823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.797587    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbmhc\" (UniqueName: \"kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-kube-api-access-mbmhc\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:13.797823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:13.797601    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402ae8c8-446f-4812-9ecc-9cb29b9c35fe-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:14.682027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:14.681990    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"
Nov 02 20:19:14.902127 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:14.902088    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"]
Nov 02 20:19:14.953533 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:14.953433    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-9p7lf"]
Nov 02 20:19:15.167445 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:15.167409    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="402ae8c8-446f-4812-9ecc-9cb29b9c35fe" path="/var/lib/kubelet/pods/402ae8c8-446f-4812-9ecc-9cb29b9c35fe/volumes"
Nov 02 20:19:15.582825 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:15.582776    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:15.752563 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:15.752529    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:16.722161 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:16.722110    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:16.898579 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:16.898540    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:19.625261 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:19.625220    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:20.480037 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:20.479995    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:21.092378 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:21.092339    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:21.509785 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:21.509671    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:22.304042 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:22.303995    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"]
Nov 02 20:19:22.601244 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:22.601148    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:22.834159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:22.834118    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:26.619349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:26.619311    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:26.800704 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:26.800656    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:26.992452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:26.992353    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:27.304987 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.304946    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs" podUID="05c7506a-5bc2-4373-8f99-644b72d4c65a" containerName="payload-processing" containerID="cri-o://d5cad01b2bf9ba5771ace176d8cc3af6ef681242bc5a0cbde4dcc532b132cc51" gracePeriod=25
Nov 02 20:19:27.359683 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:19:27.359628    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:19:27.574025 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.573958    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"
Nov 02 20:19:27.697322 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.697286    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c7506a-5bc2-4373-8f99-644b72d4c65a-kube-api-access-hq9jx\" (UniqueName: \"kubernetes.io/projected/05c7506a-5bc2-4373-8f99-644b72d4c65a-kube-api-access-hq9jx\") pod \"05c7506a-5bc2-4373-8f99-644b72d4c65a\" (UID: \"05c7506a-5bc2-4373-8f99-644b72d4c65a\") "
Nov 02 20:19:27.697869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.697439    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c7506a-5bc2-4373-8f99-644b72d4c65a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c7506a-5bc2-4373-8f99-644b72d4c65a-plugins-config-volume\") pod \"05c7506a-5bc2-4373-8f99-644b72d4c65a\" (UID: \"05c7506a-5bc2-4373-8f99-644b72d4c65a\") "
Nov 02 20:19:27.697869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.697803    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c7506a-5bc2-4373-8f99-644b72d4c65a-plugins-config-volume" pod "05c7506a-5bc2-4373-8f99-644b72d4c65a" (UID: "05c7506a-5bc2-4373-8f99-644b72d4c65a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:19:27.699783 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.699738    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c7506a-5bc2-4373-8f99-644b72d4c65a-kube-api-access-hq9jx" pod "05c7506a-5bc2-4373-8f99-644b72d4c65a" (UID: "05c7506a-5bc2-4373-8f99-644b72d4c65a"). InnerVolumeSpecName "kube-api-access-hq9jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:27.798744 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.798708    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq9jx\" (UniqueName: \"kubernetes.io/projected/05c7506a-5bc2-4373-8f99-644b72d4c65a-kube-api-access-hq9jx\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:27.798744 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.798742    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c7506a-5bc2-4373-8f99-644b72d4c65a-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:27.819582 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.819540    5224 generic.go:356] "Generic (PLEG): container finished" podID="05c7506a-5bc2-4373-8f99-644b72d4c65a" containerID="d5cad01b2bf9ba5771ace176d8cc3af6ef681242bc5a0cbde4dcc532b132cc51" exitCode=0
Nov 02 20:19:27.819762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.819620    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"
Nov 02 20:19:27.819762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.819627    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs" event={"ID":"05c7506a-5bc2-4373-8f99-644b72d4c65a","Type":"ContainerDied","Data":"d5cad01b2bf9ba5771ace176d8cc3af6ef681242bc5a0cbde4dcc532b132cc51"}
Nov 02 20:19:27.819762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.819720    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs" event={"ID":"05c7506a-5bc2-4373-8f99-644b72d4c65a","Type":"ContainerDied","Data":"687e1fda801e5728ffe3ec8d6744fd12d054177212ffb6b0f67c4d02a1e2470f"}
Nov 02 20:19:27.819932 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.819744    5224 scope.go:122] "RemoveContainer" containerID="d5cad01b2bf9ba5771ace176d8cc3af6ef681242bc5a0cbde4dcc532b132cc51"
Nov 02 20:19:27.831455 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.831432    5224 scope.go:122] "RemoveContainer" containerID="d5cad01b2bf9ba5771ace176d8cc3af6ef681242bc5a0cbde4dcc532b132cc51"
Nov 02 20:19:27.831912 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:19:27.831763    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5cad01b2bf9ba5771ace176d8cc3af6ef681242bc5a0cbde4dcc532b132cc51\": container with ID starting with d5cad01b2bf9ba5771ace176d8cc3af6ef681242bc5a0cbde4dcc532b132cc51 not found: ID does not exist" containerID="d5cad01b2bf9ba5771ace176d8cc3af6ef681242bc5a0cbde4dcc532b132cc51"
Nov 02 20:19:27.832271 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.831923    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5cad01b2bf9ba5771ace176d8cc3af6ef681242bc5a0cbde4dcc532b132cc51"} err="failed to get container status \"d5cad01b2bf9ba5771ace176d8cc3af6ef681242bc5a0cbde4dcc532b132cc51\": rpc error: code = NotFound desc = could not find container \"d5cad01b2bf9ba5771ace176d8cc3af6ef681242bc5a0cbde4dcc532b132cc51\": container with ID starting with d5cad01b2bf9ba5771ace176d8cc3af6ef681242bc5a0cbde4dcc532b132cc51 not found: ID does not exist"
Nov 02 20:19:27.989937 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.989894    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"]
Nov 02 20:19:27.992455 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.992424    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:27.993919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:27.993892    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-296ae9-865c85568f-p2fxs"]
Nov 02 20:19:29.167653 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:29.167610    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c7506a-5bc2-4373-8f99-644b72d4c65a" path="/var/lib/kubelet/pods/05c7506a-5bc2-4373-8f99-644b72d4c65a/volumes"
Nov 02 20:19:29.773236 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:29.773188    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:29.987039 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:29.987005    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:30.145850 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.145735    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"]
Nov 02 20:19:30.146256 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146242    5224 cpu_manager.go:414] "containerMap: removing container" podUID="402ae8c8-446f-4812-9ecc-9cb29b9c35fe" containerName="istio-proxy"
Nov 02 20:19:30.146307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146258    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402ae8c8-446f-4812-9ecc-9cb29b9c35fe" containerName="istio-proxy"
Nov 02 20:19:30.146307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146271    5224 cpu_manager.go:414] "containerMap: removing container" podUID="cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" containerName="istio-proxy"
Nov 02 20:19:30.146307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146276    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" containerName="istio-proxy"
Nov 02 20:19:30.146307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146285    5224 cpu_manager.go:414] "containerMap: removing container" podUID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerName="istio-proxy"
Nov 02 20:19:30.146307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146290    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerName="istio-proxy"
Nov 02 20:19:30.146460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146314    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a14b2afc-92d5-4948-aa9a-fc1d271659fe" containerName="istio-proxy"
Nov 02 20:19:30.146460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146320    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14b2afc-92d5-4948-aa9a-fc1d271659fe" containerName="istio-proxy"
Nov 02 20:19:30.146460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146326    5224 cpu_manager.go:414] "containerMap: removing container" podUID="54740835-67a3-418f-a090-18345e77f536" containerName="istio-proxy"
Nov 02 20:19:30.146460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146330    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54740835-67a3-418f-a090-18345e77f536" containerName="istio-proxy"
Nov 02 20:19:30.146460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146340    5224 cpu_manager.go:414] "containerMap: removing container" podUID="b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" containerName="istio-proxy"
Nov 02 20:19:30.146460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146344    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" containerName="istio-proxy"
Nov 02 20:19:30.146460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146360    5224 cpu_manager.go:414] "containerMap: removing container" podUID="6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" containerName="istio-proxy"
Nov 02 20:19:30.146460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146365    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" containerName="istio-proxy"
Nov 02 20:19:30.146460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146373    5224 cpu_manager.go:414] "containerMap: removing container" podUID="05c7506a-5bc2-4373-8f99-644b72d4c65a" containerName="payload-processing"
Nov 02 20:19:30.146460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146378    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c7506a-5bc2-4373-8f99-644b72d4c65a" containerName="payload-processing"
Nov 02 20:19:30.146460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146450    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerName="istio-proxy"
Nov 02 20:19:30.146460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146457    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="6529d3d9-8daa-4f3d-b0bd-3bf157236d3e" containerName="istio-proxy"
Nov 02 20:19:30.146460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146463    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c7506a-5bc2-4373-8f99-644b72d4c65a" containerName="payload-processing"
Nov 02 20:19:30.146919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146473    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="402ae8c8-446f-4812-9ecc-9cb29b9c35fe" containerName="istio-proxy"
Nov 02 20:19:30.146919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146479    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3468f2-ef1e-4554-b6c7-e5e1d6ac4d4f" containerName="istio-proxy"
Nov 02 20:19:30.146919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146485    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerName="istio-proxy"
Nov 02 20:19:30.146919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146495    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14b2afc-92d5-4948-aa9a-fc1d271659fe" containerName="istio-proxy"
Nov 02 20:19:30.146919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146501    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="54740835-67a3-418f-a090-18345e77f536" containerName="istio-proxy"
Nov 02 20:19:30.146919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.146506    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2454a23-ca3d-43d3-9a6e-3ba05db8c99f" containerName="istio-proxy"
Nov 02 20:19:30.149717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.149697    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"
Nov 02 20:19:30.202035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.201999    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"]
Nov 02 20:19:30.228836 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.228786    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfspm\" (UniqueName: \"kubernetes.io/projected/2ccad093-1904-4c78-90e8-d11c2839fe08-kube-api-access-lfspm\") pod \"payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82\" (UID: \"2ccad093-1904-4c78-90e8-d11c2839fe08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"
Nov 02 20:19:30.229042 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.228939    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ccad093-1904-4c78-90e8-d11c2839fe08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82\" (UID: \"2ccad093-1904-4c78-90e8-d11c2839fe08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"
Nov 02 20:19:30.253016 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.252982    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-316e97\""
Nov 02 20:19:30.254901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.254876    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-316e97-dockercfg-qdbwn\""
Nov 02 20:19:30.330756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.330699    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfspm\" (UniqueName: \"kubernetes.io/projected/2ccad093-1904-4c78-90e8-d11c2839fe08-kube-api-access-lfspm\") pod \"payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82\" (UID: \"2ccad093-1904-4c78-90e8-d11c2839fe08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"
Nov 02 20:19:30.330975 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.330819    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ccad093-1904-4c78-90e8-d11c2839fe08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82\" (UID: \"2ccad093-1904-4c78-90e8-d11c2839fe08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"
Nov 02 20:19:30.331928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.331904    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ccad093-1904-4c78-90e8-d11c2839fe08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82\" (UID: \"2ccad093-1904-4c78-90e8-d11c2839fe08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"
Nov 02 20:19:30.389917 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.389881    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfspm\" (UniqueName: \"kubernetes.io/projected/2ccad093-1904-4c78-90e8-d11c2839fe08-kube-api-access-lfspm\") pod \"payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82\" (UID: \"2ccad093-1904-4c78-90e8-d11c2839fe08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"
Nov 02 20:19:30.463323 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.463228    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"
Nov 02 20:19:30.832853 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:19:30.832805    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ccad093_1904_4c78_90e8_d11c2839fe08.slice/crio-cd4ea1958791204f82bb32a7cde1a6853ab3cc86db51dd585f6ab019c8d7eff6 WatchSource:0}: Error finding container cd4ea1958791204f82bb32a7cde1a6853ab3cc86db51dd585f6ab019c8d7eff6: Status 404 returned error can't find the container with id cd4ea1958791204f82bb32a7cde1a6853ab3cc86db51dd585f6ab019c8d7eff6
Nov 02 20:19:30.835294 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.835264    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"]
Nov 02 20:19:30.842629 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:30.842596    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82" event={"ID":"2ccad093-1904-4c78-90e8-d11c2839fe08","Type":"ContainerStarted","Data":"cd4ea1958791204f82bb32a7cde1a6853ab3cc86db51dd585f6ab019c8d7eff6"}
Nov 02 20:19:31.850847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:31.850805    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82" event={"ID":"2ccad093-1904-4c78-90e8-d11c2839fe08","Type":"ContainerStarted","Data":"9d1e20028c51f819a7e1be652986fac81313d117463751f98b391151408598a2"}
Nov 02 20:19:31.851282 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:31.851026    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"
Nov 02 20:19:31.968187 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:31.968131    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82" podStartSLOduration=1.96811494 podStartE2EDuration="1.96811494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:31.965449257 +0000 UTC m=+1852.963137867" watchObservedRunningTime="2026-11-02 20:19:31.96811494 +0000 UTC m=+1852.965803550"
Nov 02 20:19:32.753322 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:32.753273    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:34.984040 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:34.983998    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:36.726488 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:36.726450    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:36.906497 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:36.906459    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:38.290375 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:38.290334    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:38.719173 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:38.719065    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"]
Nov 02 20:19:38.719373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:38.719302    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22" podUID="62583a4e-72b8-407f-ac57-9baa1a1705c0" containerName="istio-proxy" containerID="cri-o://ae45c40d52e6c59e304e9b04b0a86d93bdf41e4b6c6dc3e4ac12593655315477" gracePeriod=30
Nov 02 20:19:38.730811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:38.730779    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"]
Nov 02 20:19:38.731023 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:38.731001    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562" podUID="a90fadb8-0255-48af-8269-1ff834043c25" containerName="istio-proxy" containerID="cri-o://7b8fe8b3eac0be79d9e5967c401435fb5469efd908f9f00fed8c192e6c86df80" gracePeriod=30
Nov 02 20:19:38.768704 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:38.768661    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:38.770139 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:38.770113    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"]
Nov 02 20:19:38.770347 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:38.770327    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd" podUID="94fb9c1f-b33a-4244-bd0b-ff2393f37587" containerName="istio-proxy" containerID="cri-o://393a14fd3c4d3bc5cf3a0126f28f302c77f20f5999fe933865cc7b11c3288c26" gracePeriod=30
Nov 02 20:19:38.954210 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:38.954168    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:39.342745 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:39.342707    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:39.685478 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:39.685382    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:39.901417 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:39.901384    5224 scope.go:122] "RemoveContainer" containerID="7311f2d676773f8c3f7d57d1a31fe2c419f3a25c24df218b0c40b2860b115fce"
Nov 02 20:19:40.212803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.212686    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"]
Nov 02 20:19:40.213664 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.213637    5224 cpu_manager.go:414] "containerMap: removing container" podUID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerName="istio-proxy"
Nov 02 20:19:40.213754 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.213666    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df701f2f-911b-40e0-9e88-a66aa003ee45" containerName="istio-proxy"
Nov 02 20:19:40.219233 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.219196    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.294270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.294211    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-data\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.294611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.294574    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9cf7\" (UniqueName: \"kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-kube-api-access-z9cf7\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.294935 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.294899    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-token\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.295159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.295140    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-certs\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.295345 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.295301    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-envoy\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.295551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.295534    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istiod-ca-cert\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.295745 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.295728    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-socket\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.295934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.295905    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-podinfo\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.296101 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.296072    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-credential-socket\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.342130 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.339348    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"]
Nov 02 20:19:40.351792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.348515    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e87e9a-openshift-default-dockercfg-cd4jl\""
Nov 02 20:19:40.353472 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.353358    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:40.397483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.397438    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-data\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.397721 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.397697    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9cf7\" (UniqueName: \"kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-kube-api-access-z9cf7\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.400817 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.397964    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-token\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.401130 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.401096    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-certs\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.401266 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.401251    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-envoy\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.402744 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.401392    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istiod-ca-cert\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.403001 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.402976    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-socket\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.403175 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.403159    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-podinfo\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.403326 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.403311    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-credential-socket\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.403975 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.403953    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-credential-socket\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.404111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.398119    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-data\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.404427 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.404410    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-certs\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.405794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.404988    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-socket\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.405794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.405588    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istiod-ca-cert\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.414528 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.414499    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-podinfo\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.416487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.416408    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-envoy\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.456640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.456520    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9cf7\" (UniqueName: \"kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-kube-api-access-z9cf7\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.456640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.456578    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-token\") pod \"e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") " pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.535911 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.535867    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:40.590169 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.588070    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:40.969365 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:19:40.969301    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8708cdb6_cf04_4f89_b0fc_a0a3f9fb0e0e.slice/crio-9c830e53ef36f6f315056c9398dfd980b742a17d5a184be7afa87365e547489f WatchSource:0}: Error finding container 9c830e53ef36f6f315056c9398dfd980b742a17d5a184be7afa87365e547489f: Status 404 returned error can't find the container with id 9c830e53ef36f6f315056c9398dfd980b742a17d5a184be7afa87365e547489f
Nov 02 20:19:40.972439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.972399    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:19:40.972511 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.972489    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:19:40.972553 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.972536    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:19:40.973350 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:40.973335    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"]
Nov 02 20:19:41.961135 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:41.961092    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" event={"ID":"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e","Type":"ContainerStarted","Data":"beae7518d34ac2db33d20bb221a7d303fee816ede3b0432c0a01142b1ca8abbf"}
Nov 02 20:19:41.961566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:41.961163    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" event={"ID":"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e","Type":"ContainerStarted","Data":"9c830e53ef36f6f315056c9398dfd980b742a17d5a184be7afa87365e547489f"}
Nov 02 20:19:42.181705 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:42.181653    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" podStartSLOduration=2.181637909 podStartE2EDuration="2.181637909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:42.129665797 +0000 UTC m=+1863.127354405" watchObservedRunningTime="2026-11-02 20:19:42.181637909 +0000 UTC m=+1863.179326517"
Nov 02 20:19:42.255413 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:42.255304    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:42.537009 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:42.536967    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:42.543164 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:42.543133    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:42.862688 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:42.862606    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"
Nov 02 20:19:42.967745 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:42.967709    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:42.968881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:42.968856    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:19:43.977573 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:43.977532    5224 generic.go:356] "Generic (PLEG): container finished" podID="a90fadb8-0255-48af-8269-1ff834043c25" containerID="7b8fe8b3eac0be79d9e5967c401435fb5469efd908f9f00fed8c192e6c86df80" exitCode=0
Nov 02 20:19:43.978011 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:43.977762    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562" event={"ID":"a90fadb8-0255-48af-8269-1ff834043c25","Type":"ContainerDied","Data":"7b8fe8b3eac0be79d9e5967c401435fb5469efd908f9f00fed8c192e6c86df80"}
Nov 02 20:19:43.980083 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:43.980055    5224 generic.go:356] "Generic (PLEG): container finished" podID="62583a4e-72b8-407f-ac57-9baa1a1705c0" containerID="ae45c40d52e6c59e304e9b04b0a86d93bdf41e4b6c6dc3e4ac12593655315477" exitCode=0
Nov 02 20:19:43.980226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:43.980173    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22" event={"ID":"62583a4e-72b8-407f-ac57-9baa1a1705c0","Type":"ContainerDied","Data":"ae45c40d52e6c59e304e9b04b0a86d93bdf41e4b6c6dc3e4ac12593655315477"}
Nov 02 20:19:43.982437 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:43.982410    5224 generic.go:356] "Generic (PLEG): container finished" podID="94fb9c1f-b33a-4244-bd0b-ff2393f37587" containerID="393a14fd3c4d3bc5cf3a0126f28f302c77f20f5999fe933865cc7b11c3288c26" exitCode=0
Nov 02 20:19:43.983504 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:43.983478    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd" event={"ID":"94fb9c1f-b33a-4244-bd0b-ff2393f37587","Type":"ContainerDied","Data":"393a14fd3c4d3bc5cf3a0126f28f302c77f20f5999fe933865cc7b11c3288c26"}
Nov 02 20:19:44.084061 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.084027    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:19:44.105376 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.105350    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:19:44.125024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.124992    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:19:44.261315 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261273    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-istio-token\" (UniqueName: \"kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-istio-token\") pod \"a90fadb8-0255-48af-8269-1ff834043c25\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") "
Nov 02 20:19:44.261315 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261316    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-token\" (UniqueName: \"kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-token\") pod \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") "
Nov 02 20:19:44.261588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261336    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a90fadb8-0255-48af-8269-1ff834043c25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a90fadb8-0255-48af-8269-1ff834043c25-istio-podinfo\") pod \"a90fadb8-0255-48af-8269-1ff834043c25\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") "
Nov 02 20:19:44.261588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261364    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a90fadb8-0255-48af-8269-1ff834043c25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a90fadb8-0255-48af-8269-1ff834043c25-istiod-ca-cert\") pod \"a90fadb8-0255-48af-8269-1ff834043c25\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") "
Nov 02 20:19:44.261588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261476    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-data\") pod \"62583a4e-72b8-407f-ac57-9baa1a1705c0\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") "
Nov 02 20:19:44.261588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261518    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-socket\") pod \"a90fadb8-0255-48af-8269-1ff834043c25\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") "
Nov 02 20:19:44.261588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261568    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-envoy\") pod \"a90fadb8-0255-48af-8269-1ff834043c25\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") "
Nov 02 20:19:44.261887 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261598    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-credential-socket\") pod \"a90fadb8-0255-48af-8269-1ff834043c25\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") "
Nov 02 20:19:44.261887 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261657    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-data\") pod \"a90fadb8-0255-48af-8269-1ff834043c25\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") "
Nov 02 20:19:44.261887 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261703    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-token\" (UniqueName: \"kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-token\") pod \"62583a4e-72b8-407f-ac57-9baa1a1705c0\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") "
Nov 02 20:19:44.261887 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261800    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-credential-socket\") pod \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") "
Nov 02 20:19:44.261887 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261807    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90fadb8-0255-48af-8269-1ff834043c25-istiod-ca-cert" pod "a90fadb8-0255-48af-8269-1ff834043c25" (UID: "a90fadb8-0255-48af-8269-1ff834043c25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:19:44.261887 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261837    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-certs\") pod \"62583a4e-72b8-407f-ac57-9baa1a1705c0\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") "
Nov 02 20:19:44.261887 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261879    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-credential-socket\") pod \"62583a4e-72b8-407f-ac57-9baa1a1705c0\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") "
Nov 02 20:19:44.262253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261914    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-data\") pod \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") "
Nov 02 20:19:44.262253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.261975    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-socket\") pod \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") "
Nov 02 20:19:44.262253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262019    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-envoy\") pod \"62583a4e-72b8-407f-ac57-9baa1a1705c0\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") "
Nov 02 20:19:44.262253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262063    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-certs\") pod \"a90fadb8-0255-48af-8269-1ff834043c25\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") "
Nov 02 20:19:44.262253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262074    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-data" pod "62583a4e-72b8-407f-ac57-9baa1a1705c0" (UID: "62583a4e-72b8-407f-ac57-9baa1a1705c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.262253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262097    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-envoy\") pod \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") "
Nov 02 20:19:44.262253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262162    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-podinfo\") pod \"62583a4e-72b8-407f-ac57-9baa1a1705c0\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") "
Nov 02 20:19:44.262253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262209    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-kube-api-access-g4xdh\" (UniqueName: \"kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-kube-api-access-g4xdh\") pod \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") "
Nov 02 20:19:44.262253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262248    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-certs" pod "62583a4e-72b8-407f-ac57-9baa1a1705c0" (UID: "62583a4e-72b8-407f-ac57-9baa1a1705c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.262717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262263    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-socket\") pod \"62583a4e-72b8-407f-ac57-9baa1a1705c0\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") "
Nov 02 20:19:44.262717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262298    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-kube-api-access-kdj28\" (UniqueName: \"kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-kube-api-access-kdj28\") pod \"62583a4e-72b8-407f-ac57-9baa1a1705c0\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") "
Nov 02 20:19:44.262717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262344    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-podinfo\") pod \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") "
Nov 02 20:19:44.262717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262420    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-credential-socket" pod "a90fadb8-0255-48af-8269-1ff834043c25" (UID: "a90fadb8-0255-48af-8269-1ff834043c25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.262717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262424    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istiod-ca-cert\") pod \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") "
Nov 02 20:19:44.262717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262473    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62583a4e-72b8-407f-ac57-9baa1a1705c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62583a4e-72b8-407f-ac57-9baa1a1705c0-istiod-ca-cert\") pod \"62583a4e-72b8-407f-ac57-9baa1a1705c0\" (UID: \"62583a4e-72b8-407f-ac57-9baa1a1705c0\") "
Nov 02 20:19:44.262717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262532    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-certs\") pod \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\" (UID: \"94fb9c1f-b33a-4244-bd0b-ff2393f37587\") "
Nov 02 20:19:44.262717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262566    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-kube-api-access-gpcnv\" (UniqueName: \"kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-kube-api-access-gpcnv\") pod \"a90fadb8-0255-48af-8269-1ff834043c25\" (UID: \"a90fadb8-0255-48af-8269-1ff834043c25\") "
Nov 02 20:19:44.262717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.262576    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-data" pod "a90fadb8-0255-48af-8269-1ff834043c25" (UID: "a90fadb8-0255-48af-8269-1ff834043c25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.263460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.263419    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istiod-ca-cert" pod "94fb9c1f-b33a-4244-bd0b-ff2393f37587" (UID: "94fb9c1f-b33a-4244-bd0b-ff2393f37587"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:19:44.263803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.263734    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-socket" pod "62583a4e-72b8-407f-ac57-9baa1a1705c0" (UID: "62583a4e-72b8-407f-ac57-9baa1a1705c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.265372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.264760    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-credential-socket" pod "62583a4e-72b8-407f-ac57-9baa1a1705c0" (UID: "62583a4e-72b8-407f-ac57-9baa1a1705c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.265372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.265027    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.265372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.265056    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.265372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.265072    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.265372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.265077    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a90fadb8-0255-48af-8269-1ff834043c25-istio-podinfo" pod "a90fadb8-0255-48af-8269-1ff834043c25" (UID: "a90fadb8-0255-48af-8269-1ff834043c25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:19:44.265372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.265092    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.265372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.265105    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.265372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.265119    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.265372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.265133    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a90fadb8-0255-48af-8269-1ff834043c25-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.265372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.265148    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.265962 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.265538    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-socket" pod "a90fadb8-0255-48af-8269-1ff834043c25" (UID: "a90fadb8-0255-48af-8269-1ff834043c25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.266615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.266084    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-certs" pod "94fb9c1f-b33a-4244-bd0b-ff2393f37587" (UID: "94fb9c1f-b33a-4244-bd0b-ff2393f37587"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.266615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.266256    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62583a4e-72b8-407f-ac57-9baa1a1705c0-istiod-ca-cert" pod "62583a4e-72b8-407f-ac57-9baa1a1705c0" (UID: "62583a4e-72b8-407f-ac57-9baa1a1705c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:19:44.266615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.266296    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-socket" pod "94fb9c1f-b33a-4244-bd0b-ff2393f37587" (UID: "94fb9c1f-b33a-4244-bd0b-ff2393f37587"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.266615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.266328    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-certs" pod "a90fadb8-0255-48af-8269-1ff834043c25" (UID: "a90fadb8-0255-48af-8269-1ff834043c25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.266615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.266592    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-istio-token" pod "a90fadb8-0255-48af-8269-1ff834043c25" (UID: "a90fadb8-0255-48af-8269-1ff834043c25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:44.267009 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.266617    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-credential-socket" pod "94fb9c1f-b33a-4244-bd0b-ff2393f37587" (UID: "94fb9c1f-b33a-4244-bd0b-ff2393f37587"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.267009 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.266630    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-token" pod "62583a4e-72b8-407f-ac57-9baa1a1705c0" (UID: "62583a4e-72b8-407f-ac57-9baa1a1705c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:44.267009 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.266647    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-data" pod "94fb9c1f-b33a-4244-bd0b-ff2393f37587" (UID: "94fb9c1f-b33a-4244-bd0b-ff2393f37587"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.267009 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.266759    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-kube-api-access-g4xdh" pod "94fb9c1f-b33a-4244-bd0b-ff2393f37587" (UID: "94fb9c1f-b33a-4244-bd0b-ff2393f37587"). InnerVolumeSpecName "kube-api-access-g4xdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:44.267288 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.267063    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-envoy" pod "a90fadb8-0255-48af-8269-1ff834043c25" (UID: "a90fadb8-0255-48af-8269-1ff834043c25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.267288 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.267199    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-kube-api-access-kdj28" pod "62583a4e-72b8-407f-ac57-9baa1a1705c0" (UID: "62583a4e-72b8-407f-ac57-9baa1a1705c0"). InnerVolumeSpecName "kube-api-access-kdj28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:44.267511 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.267478    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-token" pod "94fb9c1f-b33a-4244-bd0b-ff2393f37587" (UID: "94fb9c1f-b33a-4244-bd0b-ff2393f37587"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:44.267731 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.267683    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-kube-api-access-gpcnv" pod "a90fadb8-0255-48af-8269-1ff834043c25" (UID: "a90fadb8-0255-48af-8269-1ff834043c25"). InnerVolumeSpecName "kube-api-access-gpcnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:44.267873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.267837    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-podinfo" pod "94fb9c1f-b33a-4244-bd0b-ff2393f37587" (UID: "94fb9c1f-b33a-4244-bd0b-ff2393f37587"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:19:44.268034 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.268017    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-envoy" pod "94fb9c1f-b33a-4244-bd0b-ff2393f37587" (UID: "94fb9c1f-b33a-4244-bd0b-ff2393f37587"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.268895 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.268863    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-podinfo" pod "62583a4e-72b8-407f-ac57-9baa1a1705c0" (UID: "62583a4e-72b8-407f-ac57-9baa1a1705c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:19:44.269435 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.269408    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-envoy" pod "62583a4e-72b8-407f-ac57-9baa1a1705c0" (UID: "62583a4e-72b8-407f-ac57-9baa1a1705c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:44.366302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366262    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366305    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpcnv\" (UniqueName: \"kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-kube-api-access-gpcnv\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366322    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a90fadb8-0255-48af-8269-1ff834043c25-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366337    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366351    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a90fadb8-0255-48af-8269-1ff834043c25-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366365    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366379    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366393    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366421    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366435    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366449    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366463    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366483    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a90fadb8-0255-48af-8269-1ff834043c25-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366497    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366510    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62583a4e-72b8-407f-ac57-9baa1a1705c0-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366523    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4xdh\" (UniqueName: \"kubernetes.io/projected/94fb9c1f-b33a-4244-bd0b-ff2393f37587-kube-api-access-g4xdh\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366538    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdj28\" (UniqueName: \"kubernetes.io/projected/62583a4e-72b8-407f-ac57-9baa1a1705c0-kube-api-access-kdj28\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366554    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94fb9c1f-b33a-4244-bd0b-ff2393f37587-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.366901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.366571    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62583a4e-72b8-407f-ac57-9baa1a1705c0-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:44.944173 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.944135    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:44.990700 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.990666    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"
Nov 02 20:19:44.991232 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.990694    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562" event={"ID":"a90fadb8-0255-48af-8269-1ff834043c25","Type":"ContainerDied","Data":"d51afef576437a85ce581d796acf1756a759ce233dfb1e9ac637da0a99d4bc70"}
Nov 02 20:19:44.991232 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.990802    5224 scope.go:122] "RemoveContainer" containerID="7b8fe8b3eac0be79d9e5967c401435fb5469efd908f9f00fed8c192e6c86df80"
Nov 02 20:19:44.992368 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.992344    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"
Nov 02 20:19:44.992368 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.992365    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22" event={"ID":"62583a4e-72b8-407f-ac57-9baa1a1705c0","Type":"ContainerDied","Data":"2ee03eaf9b1dc96197b0af7ddc41d1ae130aebbef77e1f7cff5d8cf69685cfe6"}
Nov 02 20:19:45.004259 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:44.995633    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"
Nov 02 20:19:45.004259 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:45.001066    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd" event={"ID":"94fb9c1f-b33a-4244-bd0b-ff2393f37587","Type":"ContainerDied","Data":"78673ec2bcddb2fe275563dd4c159b19a6bb0d7a84fb6c9f1434936e87ec75da"}
Nov 02 20:19:45.023269 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:45.023242    5224 scope.go:122] "RemoveContainer" containerID="ae45c40d52e6c59e304e9b04b0a86d93bdf41e4b6c6dc3e4ac12593655315477"
Nov 02 20:19:45.032977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:45.032959    5224 scope.go:122] "RemoveContainer" containerID="393a14fd3c4d3bc5cf3a0126f28f302c77f20f5999fe933865cc7b11c3288c26"
Nov 02 20:19:45.135078 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:45.135042    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:45.167459 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:45.167428    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"]
Nov 02 20:19:45.214617 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:45.214580    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-cb562"]
Nov 02 20:19:45.279168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:45.279132    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"]
Nov 02 20:19:45.282329 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:45.282303    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-m6hrd"]
Nov 02 20:19:45.317072 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:45.317028    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:45.394749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:45.394700    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"]
Nov 02 20:19:45.397238 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:45.397208    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-sdh22"]
Nov 02 20:19:47.168323 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:47.168287    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62583a4e-72b8-407f-ac57-9baa1a1705c0" path="/var/lib/kubelet/pods/62583a4e-72b8-407f-ac57-9baa1a1705c0/volumes"
Nov 02 20:19:47.168822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:47.168807    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94fb9c1f-b33a-4244-bd0b-ff2393f37587" path="/var/lib/kubelet/pods/94fb9c1f-b33a-4244-bd0b-ff2393f37587/volumes"
Nov 02 20:19:47.169273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:47.169258    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a90fadb8-0255-48af-8269-1ff834043c25" path="/var/lib/kubelet/pods/a90fadb8-0255-48af-8269-1ff834043c25/volumes"
Nov 02 20:19:47.664908 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:47.664871    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:48.098203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.098168    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:48.272366 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.272326    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:48.456324 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.456237    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"]
Nov 02 20:19:48.456906 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.456876    5224 cpu_manager.go:414] "containerMap: removing container" podUID="62583a4e-72b8-407f-ac57-9baa1a1705c0" containerName="istio-proxy"
Nov 02 20:19:48.456906 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.456900    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62583a4e-72b8-407f-ac57-9baa1a1705c0" containerName="istio-proxy"
Nov 02 20:19:48.457129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.456928    5224 cpu_manager.go:414] "containerMap: removing container" podUID="94fb9c1f-b33a-4244-bd0b-ff2393f37587" containerName="istio-proxy"
Nov 02 20:19:48.457129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.456935    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fb9c1f-b33a-4244-bd0b-ff2393f37587" containerName="istio-proxy"
Nov 02 20:19:48.457129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.456967    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a90fadb8-0255-48af-8269-1ff834043c25" containerName="istio-proxy"
Nov 02 20:19:48.457129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.456973    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a90fadb8-0255-48af-8269-1ff834043c25" containerName="istio-proxy"
Nov 02 20:19:48.457129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.457065    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a90fadb8-0255-48af-8269-1ff834043c25" containerName="istio-proxy"
Nov 02 20:19:48.457129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.457077    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="62583a4e-72b8-407f-ac57-9baa1a1705c0" containerName="istio-proxy"
Nov 02 20:19:48.457129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.457087    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fb9c1f-b33a-4244-bd0b-ff2393f37587" containerName="istio-proxy"
Nov 02 20:19:48.462235 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.462209    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"
Nov 02 20:19:48.513962 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.513924    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"]
Nov 02 20:19:48.519240 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.519194    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpjv9\" (UniqueName: \"kubernetes.io/projected/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-kube-api-access-gpjv9\") pod \"payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf\" (UID: \"80860d6f-ec62-427a-a99f-a6b4c4ff3c18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"
Nov 02 20:19:48.520673 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.520644    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf\" (UID: \"80860d6f-ec62-427a-a99f-a6b4c4ff3c18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"
Nov 02 20:19:48.579806 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.577121    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e87e9a-dockercfg-26lv4\""
Nov 02 20:19:48.579806 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.577246    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e87e9a\""
Nov 02 20:19:48.636300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.628811    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf\" (UID: \"80860d6f-ec62-427a-a99f-a6b4c4ff3c18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"
Nov 02 20:19:48.636300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.629063    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpjv9\" (UniqueName: \"kubernetes.io/projected/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-kube-api-access-gpjv9\") pod \"payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf\" (UID: \"80860d6f-ec62-427a-a99f-a6b4c4ff3c18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"
Nov 02 20:19:48.636300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.630284    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf\" (UID: \"80860d6f-ec62-427a-a99f-a6b4c4ff3c18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"
Nov 02 20:19:48.636300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.632363    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"]
Nov 02 20:19:48.644792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.642323    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"
Nov 02 20:19:48.688059 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.688019    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"]
Nov 02 20:19:48.690012 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.689980    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpjv9\" (UniqueName: \"kubernetes.io/projected/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-kube-api-access-gpjv9\") pod \"payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf\" (UID: \"80860d6f-ec62-427a-a99f-a6b4c4ff3c18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"
Nov 02 20:19:48.730488 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.730390    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hj56\" (UniqueName: \"kubernetes.io/projected/23fae0cb-a909-420d-9337-aa80cd97fdb1-kube-api-access-6hj56\") pod \"payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj\" (UID: \"23fae0cb-a909-420d-9337-aa80cd97fdb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"
Nov 02 20:19:48.730488 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.730461    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fae0cb-a909-420d-9337-aa80cd97fdb1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj\" (UID: \"23fae0cb-a909-420d-9337-aa80cd97fdb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"
Nov 02 20:19:48.776134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.776091    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"
Nov 02 20:19:48.834401 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.834370    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hj56\" (UniqueName: \"kubernetes.io/projected/23fae0cb-a909-420d-9337-aa80cd97fdb1-kube-api-access-6hj56\") pod \"payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj\" (UID: \"23fae0cb-a909-420d-9337-aa80cd97fdb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"
Nov 02 20:19:48.834540 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.834463    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fae0cb-a909-420d-9337-aa80cd97fdb1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj\" (UID: \"23fae0cb-a909-420d-9337-aa80cd97fdb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"
Nov 02 20:19:48.835434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.835406    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fae0cb-a909-420d-9337-aa80cd97fdb1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj\" (UID: \"23fae0cb-a909-420d-9337-aa80cd97fdb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"
Nov 02 20:19:48.893185 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.893148    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hj56\" (UniqueName: \"kubernetes.io/projected/23fae0cb-a909-420d-9337-aa80cd97fdb1-kube-api-access-6hj56\") pod \"payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj\" (UID: \"23fae0cb-a909-420d-9337-aa80cd97fdb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"
Nov 02 20:19:48.969439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:48.969404    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"
Nov 02 20:19:49.153363 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:19:49.153331    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80860d6f_ec62_427a_a99f_a6b4c4ff3c18.slice/crio-dd35b20fdec16fd344a099eb5f84ced175a82499faacece0201086f549427de7 WatchSource:0}: Error finding container dd35b20fdec16fd344a099eb5f84ced175a82499faacece0201086f549427de7: Status 404 returned error can't find the container with id dd35b20fdec16fd344a099eb5f84ced175a82499faacece0201086f549427de7
Nov 02 20:19:49.157117 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:49.157094    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"]
Nov 02 20:19:49.339317 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:19:49.339285    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fae0cb_a909_420d_9337_aa80cd97fdb1.slice/crio-f209d10ef47d0e75c794081ea48419c1ba83a20b354df8c44b9a46ef0bb89e0f WatchSource:0}: Error finding container f209d10ef47d0e75c794081ea48419c1ba83a20b354df8c44b9a46ef0bb89e0f: Status 404 returned error can't find the container with id f209d10ef47d0e75c794081ea48419c1ba83a20b354df8c44b9a46ef0bb89e0f
Nov 02 20:19:49.341813 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:49.341790    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"]
Nov 02 20:19:50.035311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:50.035273    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf" event={"ID":"80860d6f-ec62-427a-a99f-a6b4c4ff3c18","Type":"ContainerStarted","Data":"61a27badd2e7fe7c7ce22e21bace7f7eb5113420a2656ec336b1e73389f017d2"}
Nov 02 20:19:50.035495 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:50.035330    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf" event={"ID":"80860d6f-ec62-427a-a99f-a6b4c4ff3c18","Type":"ContainerStarted","Data":"dd35b20fdec16fd344a099eb5f84ced175a82499faacece0201086f549427de7"}
Nov 02 20:19:50.035495 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:50.035458    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"
Nov 02 20:19:50.037090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:50.037061    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj" event={"ID":"23fae0cb-a909-420d-9337-aa80cd97fdb1","Type":"ContainerStarted","Data":"862308ef211fb661894cf2afad36eba471b395244d372a51f5898ba66aebcb03"}
Nov 02 20:19:50.037203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:50.037108    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj" event={"ID":"23fae0cb-a909-420d-9337-aa80cd97fdb1","Type":"ContainerStarted","Data":"f209d10ef47d0e75c794081ea48419c1ba83a20b354df8c44b9a46ef0bb89e0f"}
Nov 02 20:19:50.037203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:50.037129    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"
Nov 02 20:19:50.152906 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:50.152845    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf" podStartSLOduration=2.152829885 podStartE2EDuration="2.152829885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:50.151156342 +0000 UTC m=+1871.148844950" watchObservedRunningTime="2026-11-02 20:19:50.152829885 +0000 UTC m=+1871.150518493"
Nov 02 20:19:50.268895 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:50.268821    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj" podStartSLOduration=2.268798106 podStartE2EDuration="2.268798106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:50.265214366 +0000 UTC m=+1871.262902976" watchObservedRunningTime="2026-11-02 20:19:50.268798106 +0000 UTC m=+1871.266486716"
Nov 02 20:19:51.677446 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:51.677406    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:52.222145 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:52.222102    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:53.230749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.230708    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"]
Nov 02 20:19:53.235103 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.235075    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"
Nov 02 20:19:53.287563 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.287526    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"]
Nov 02 20:19:53.293707 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.293668    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6mf4\" (UniqueName: \"kubernetes.io/projected/b5fd73b1-9a09-4db1-8d06-c56bd893a471-kube-api-access-t6mf4\") pod \"payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz\" (UID: \"b5fd73b1-9a09-4db1-8d06-c56bd893a471\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"
Nov 02 20:19:53.293997 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.293966    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fd73b1-9a09-4db1-8d06-c56bd893a471-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz\" (UID: \"b5fd73b1-9a09-4db1-8d06-c56bd893a471\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"
Nov 02 20:19:53.335542 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.335508    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a999d7\""
Nov 02 20:19:53.335804 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.335637    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a999d7-dockercfg-mt4zc\""
Nov 02 20:19:53.395711 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.395665    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6mf4\" (UniqueName: \"kubernetes.io/projected/b5fd73b1-9a09-4db1-8d06-c56bd893a471-kube-api-access-t6mf4\") pod \"payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz\" (UID: \"b5fd73b1-9a09-4db1-8d06-c56bd893a471\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"
Nov 02 20:19:53.395928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.395889    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fd73b1-9a09-4db1-8d06-c56bd893a471-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz\" (UID: \"b5fd73b1-9a09-4db1-8d06-c56bd893a471\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"
Nov 02 20:19:53.396681 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.396658    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fd73b1-9a09-4db1-8d06-c56bd893a471-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz\" (UID: \"b5fd73b1-9a09-4db1-8d06-c56bd893a471\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"
Nov 02 20:19:53.454097 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.454057    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6mf4\" (UniqueName: \"kubernetes.io/projected/b5fd73b1-9a09-4db1-8d06-c56bd893a471-kube-api-access-t6mf4\") pod \"payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz\" (UID: \"b5fd73b1-9a09-4db1-8d06-c56bd893a471\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"
Nov 02 20:19:53.551818 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.551753    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"
Nov 02 20:19:53.936746 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:19:53.936710    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fd73b1_9a09_4db1_8d06_c56bd893a471.slice/crio-18e0a5f3f95fee319ae382eaa6940e1c93f9bee3209b8b91c6cbc773d648239d WatchSource:0}: Error finding container 18e0a5f3f95fee319ae382eaa6940e1c93f9bee3209b8b91c6cbc773d648239d: Status 404 returned error can't find the container with id 18e0a5f3f95fee319ae382eaa6940e1c93f9bee3209b8b91c6cbc773d648239d
Nov 02 20:19:53.938914 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.938895    5224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:19:53.941416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.941390    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"]
Nov 02 20:19:53.994794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:53.994746    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:19:54.066082 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:54.066042    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz" event={"ID":"b5fd73b1-9a09-4db1-8d06-c56bd893a471","Type":"ContainerStarted","Data":"18e0a5f3f95fee319ae382eaa6940e1c93f9bee3209b8b91c6cbc773d648239d"}
Nov 02 20:19:55.158684 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:55.158641    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:55.369360 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:55.369319    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:56.023410 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:56.023368    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:19:57.088421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:57.088384    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz" event={"ID":"b5fd73b1-9a09-4db1-8d06-c56bd893a471","Type":"ContainerStarted","Data":"0e45d2d87877ff2d8c2c5804080d8c0a16057237d3b2f3c323bca035d97de704"}
Nov 02 20:19:57.088910 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:57.088497    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"
Nov 02 20:19:57.206232 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:19:57.206161    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz" podStartSLOduration=1.812220779 podStartE2EDuration="4.206144767s" totalImagesPullingTime="2.393923988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:57.204130704 +0000 UTC m=+1878.201819313" watchObservedRunningTime="2026-11-02 20:19:57.206144767 +0000 UTC m=+1878.203833375"
Nov 02 20:19:57.369318 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:19:57.369228    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:20:00.219097 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:00.219057    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:01.047301 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:01.047269    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"
Nov 02 20:20:01.047497 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:01.047355    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"
Nov 02 20:20:03.203623 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:03.203577    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:03.777072 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:03.777016    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:04.180329 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:04.180228    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:04.983705 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:04.983666    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:05.197787 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:05.197737    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:07.835541 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:07.835500    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:08.016644 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:08.016606    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:08.110054 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:08.109943    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"
Nov 02 20:20:08.200810 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:08.200736    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:10.618865 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:10.618829    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:11.374804 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.374755    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"]
Nov 02 20:20:11.378819 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.378795    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"
Nov 02 20:20:11.430337 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.430303    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"]
Nov 02 20:20:11.432352 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.432308    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq\" (UID: \"8b53b416-4e46-4fee-9ebc-65f13cfb0f36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"
Nov 02 20:20:11.432541 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.432358    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmdqc\" (UniqueName: \"kubernetes.io/projected/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-kube-api-access-wmdqc\") pod \"payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq\" (UID: \"8b53b416-4e46-4fee-9ebc-65f13cfb0f36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"
Nov 02 20:20:11.479043 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.479000    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3b8db2\""
Nov 02 20:20:11.542862 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.542821    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq\" (UID: \"8b53b416-4e46-4fee-9ebc-65f13cfb0f36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"
Nov 02 20:20:11.546907 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.541883    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq\" (UID: \"8b53b416-4e46-4fee-9ebc-65f13cfb0f36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"
Nov 02 20:20:11.547152 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.547133    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdqc\" (UniqueName: \"kubernetes.io/projected/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-kube-api-access-wmdqc\") pod \"payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq\" (UID: \"8b53b416-4e46-4fee-9ebc-65f13cfb0f36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"
Nov 02 20:20:11.559425 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.559368    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"]
Nov 02 20:20:11.573483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.573447    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"
Nov 02 20:20:11.592845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.592805    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"]
Nov 02 20:20:11.605665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.605632    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdqc\" (UniqueName: \"kubernetes.io/projected/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-kube-api-access-wmdqc\") pod \"payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq\" (UID: \"8b53b416-4e46-4fee-9ebc-65f13cfb0f36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"
Nov 02 20:20:11.627553 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.627463    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3b8db2-dockercfg-jdcpr\""
Nov 02 20:20:11.655655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.655620    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh52j\" (UniqueName: \"kubernetes.io/projected/e8d32f2b-b566-4c09-803c-8b35ef3aca06-kube-api-access-bh52j\") pod \"payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm\" (UID: \"e8d32f2b-b566-4c09-803c-8b35ef3aca06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"
Nov 02 20:20:11.655907 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.655759    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d32f2b-b566-4c09-803c-8b35ef3aca06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm\" (UID: \"e8d32f2b-b566-4c09-803c-8b35ef3aca06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"
Nov 02 20:20:11.693514 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.693475    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"
Nov 02 20:20:11.757158 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.757114    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh52j\" (UniqueName: \"kubernetes.io/projected/e8d32f2b-b566-4c09-803c-8b35ef3aca06-kube-api-access-bh52j\") pod \"payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm\" (UID: \"e8d32f2b-b566-4c09-803c-8b35ef3aca06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"
Nov 02 20:20:11.757305 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.757252    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d32f2b-b566-4c09-803c-8b35ef3aca06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm\" (UID: \"e8d32f2b-b566-4c09-803c-8b35ef3aca06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"
Nov 02 20:20:11.758276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.758247    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d32f2b-b566-4c09-803c-8b35ef3aca06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm\" (UID: \"e8d32f2b-b566-4c09-803c-8b35ef3aca06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"
Nov 02 20:20:11.816727 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.816687    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh52j\" (UniqueName: \"kubernetes.io/projected/e8d32f2b-b566-4c09-803c-8b35ef3aca06-kube-api-access-bh52j\") pod \"payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm\" (UID: \"e8d32f2b-b566-4c09-803c-8b35ef3aca06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"
Nov 02 20:20:11.893679 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:11.893582    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"
Nov 02 20:20:12.063353 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:20:12.063303    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b53b416_4e46_4fee_9ebc_65f13cfb0f36.slice/crio-bf13db886b0266065480def5cb6f48c8e286e1074893363f3b4b8ebeff196b13 WatchSource:0}: Error finding container bf13db886b0266065480def5cb6f48c8e286e1074893363f3b4b8ebeff196b13: Status 404 returned error can't find the container with id bf13db886b0266065480def5cb6f48c8e286e1074893363f3b4b8ebeff196b13
Nov 02 20:20:12.064200 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:12.064163    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:12.066521 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:12.066497    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"]
Nov 02 20:20:12.185596 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:12.185504    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq" event={"ID":"8b53b416-4e46-4fee-9ebc-65f13cfb0f36","Type":"ContainerStarted","Data":"c70a5fc2ba962493d27f262500860eb1b49b7be02195c3d899d008b8dfe509bc"}
Nov 02 20:20:12.185799 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:12.185698    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"
Nov 02 20:20:12.185799 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:12.185743    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq" event={"ID":"8b53b416-4e46-4fee-9ebc-65f13cfb0f36","Type":"ContainerStarted","Data":"bf13db886b0266065480def5cb6f48c8e286e1074893363f3b4b8ebeff196b13"}
Nov 02 20:20:12.253932 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:12.253895    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:12.265569 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:20:12.265537    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d32f2b_b566_4c09_803c_8b35ef3aca06.slice/crio-b8a6869c3fa09299a67bc170fa3844ea60f34c43a338ba26fee78605fa5b1e72 WatchSource:0}: Error finding container b8a6869c3fa09299a67bc170fa3844ea60f34c43a338ba26fee78605fa5b1e72: Status 404 returned error can't find the container with id b8a6869c3fa09299a67bc170fa3844ea60f34c43a338ba26fee78605fa5b1e72
Nov 02 20:20:12.269097 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:12.269071    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"]
Nov 02 20:20:12.302736 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:12.302681    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq" podStartSLOduration=1.302665 podStartE2EDuration="1.302665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:12.299856831 +0000 UTC m=+1893.297545443" watchObservedRunningTime="2026-11-02 20:20:12.302665 +0000 UTC m=+1893.300353610"
Nov 02 20:20:13.194096 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:13.194059    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm" event={"ID":"e8d32f2b-b566-4c09-803c-8b35ef3aca06","Type":"ContainerStarted","Data":"b824121c944d5a4479d59c29a86c3dfc6d6eb33c0bc6b5658025732c0b588f23"}
Nov 02 20:20:13.194533 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:13.194126    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm" event={"ID":"e8d32f2b-b566-4c09-803c-8b35ef3aca06","Type":"ContainerStarted","Data":"b8a6869c3fa09299a67bc170fa3844ea60f34c43a338ba26fee78605fa5b1e72"}
Nov 02 20:20:14.200331 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:14.200293    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"
Nov 02 20:20:14.613377 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:14.613334    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:15.160609 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:15.160563    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:16.948217 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:16.948152    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm" podStartSLOduration=5.948135001 podStartE2EDuration="5.948135001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:13.362352153 +0000 UTC m=+1894.360040818" watchObservedRunningTime="2026-11-02 20:20:16.948135001 +0000 UTC m=+1897.945823655"
Nov 02 20:20:16.949616 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:16.949593    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"]
Nov 02 20:20:16.953941 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:16.953920    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"
Nov 02 20:20:17.008007 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:17.007966    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"]
Nov 02 20:20:17.031665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:17.031618    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmbpr\" (UniqueName: \"kubernetes.io/projected/9eb40531-747d-4ee6-99d3-cf76163f4b83-kube-api-access-cmbpr\") pod \"payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl\" (UID: \"9eb40531-747d-4ee6-99d3-cf76163f4b83\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"
Nov 02 20:20:17.031886 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:17.031793    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb40531-747d-4ee6-99d3-cf76163f4b83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl\" (UID: \"9eb40531-747d-4ee6-99d3-cf76163f4b83\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"
Nov 02 20:20:17.056895 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:17.056851    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-296ae9\""
Nov 02 20:20:17.057096 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:17.056934    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-296ae9-dockercfg-94g4m\""
Nov 02 20:20:17.133627 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:17.133588    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmbpr\" (UniqueName: \"kubernetes.io/projected/9eb40531-747d-4ee6-99d3-cf76163f4b83-kube-api-access-cmbpr\") pod \"payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl\" (UID: \"9eb40531-747d-4ee6-99d3-cf76163f4b83\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"
Nov 02 20:20:17.133865 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:17.133654    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb40531-747d-4ee6-99d3-cf76163f4b83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl\" (UID: \"9eb40531-747d-4ee6-99d3-cf76163f4b83\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"
Nov 02 20:20:17.134477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:17.134450    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb40531-747d-4ee6-99d3-cf76163f4b83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl\" (UID: \"9eb40531-747d-4ee6-99d3-cf76163f4b83\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"
Nov 02 20:20:17.193182 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:17.193149    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmbpr\" (UniqueName: \"kubernetes.io/projected/9eb40531-747d-4ee6-99d3-cf76163f4b83-kube-api-access-cmbpr\") pod \"payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl\" (UID: \"9eb40531-747d-4ee6-99d3-cf76163f4b83\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"
Nov 02 20:20:17.269379 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:17.269341    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"
Nov 02 20:20:17.646608 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:17.646562    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"]
Nov 02 20:20:18.107930 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:18.107824    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:18.228578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:18.228536    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl" event={"ID":"9eb40531-747d-4ee6-99d3-cf76163f4b83","Type":"ContainerStarted","Data":"9122be1fd785b0dd7bef145df58280d834647b37b0c5025d23dad2b0976c1100"}
Nov 02 20:20:18.228822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:18.228610    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"
Nov 02 20:20:18.228822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:18.228637    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl" event={"ID":"9eb40531-747d-4ee6-99d3-cf76163f4b83","Type":"ContainerStarted","Data":"576f3681973a3af7dab4a58955d0d0f6f60df792349b1187f6246b287894610f"}
Nov 02 20:20:18.350648 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:18.350588    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl" podStartSLOduration=2.350568999 podStartE2EDuration="2.350568999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:18.34721218 +0000 UTC m=+1899.344900789" watchObservedRunningTime="2026-11-02 20:20:18.350568999 +0000 UTC m=+1899.348257609"
Nov 02 20:20:23.197833 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:23.197797    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"
Nov 02 20:20:23.650008 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:23.649965    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:25.211492 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:25.211463    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"
Nov 02 20:20:27.343706 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:27.343665    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:27.361605 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:20:27.361566    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:20:27.569523 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:27.569485    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:28.408436 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:28.408390    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:28.585399 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:28.585359    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:28.754031 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:28.753927    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:29.172213 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:29.172175    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:29.237721 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:29.237682    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"
Nov 02 20:20:32.798829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:32.798780    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:33.560668 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:33.560630    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:34.945314 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:34.945274    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:38.573343 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:38.573307    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:39.950900 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:39.950864    5224 scope.go:122] "RemoveContainer" containerID="445185083beb114ddc71bae93203b6d26195587782c4da87466863e0a66a246f"
Nov 02 20:20:40.795013 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:40.794972    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"]
Nov 02 20:20:40.795278 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:40.795234    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" podUID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" containerName="istio-proxy" containerID="cri-o://1ab9e3d0087e5d676e0a29cd44e0bebd66bd2c42f456d8818c426e3b044b8b14" gracePeriod=30
Nov 02 20:20:45.113131 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:45.113081    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:20:45.114843 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:45.113182    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" podUID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:20:45.446613 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:45.446496    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:45.892708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:45.892653    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:20:46.089505 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.089469    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:20:46.214597 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.214558    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-data\") pod \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") "
Nov 02 20:20:46.215123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.214613    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istiod-ca-cert\") pod \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") "
Nov 02 20:20:46.215123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.214650    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-socket\") pod \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") "
Nov 02 20:20:46.215123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.214720    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-credential-socket\") pod \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") "
Nov 02 20:20:46.215123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.214831    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-podinfo\") pod \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") "
Nov 02 20:20:46.215123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.214867    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-envoy\") pod \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") "
Nov 02 20:20:46.215123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.214900    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-kube-api-access-d9pvm\" (UniqueName: \"kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-kube-api-access-d9pvm\") pod \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") "
Nov 02 20:20:46.215123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.214988    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-data" pod "6a0279bf-c6af-4283-9db8-60d0c8dd7a44" (UID: "6a0279bf-c6af-4283-9db8-60d0c8dd7a44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:46.215123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.215032    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-certs\") pod \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") "
Nov 02 20:20:46.215123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.215056    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-socket" pod "6a0279bf-c6af-4283-9db8-60d0c8dd7a44" (UID: "6a0279bf-c6af-4283-9db8-60d0c8dd7a44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:46.215123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.215084    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-token\" (UniqueName: \"kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-token\") pod \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\" (UID: \"6a0279bf-c6af-4283-9db8-60d0c8dd7a44\") "
Nov 02 20:20:46.215716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.215141    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istiod-ca-cert" pod "6a0279bf-c6af-4283-9db8-60d0c8dd7a44" (UID: "6a0279bf-c6af-4283-9db8-60d0c8dd7a44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:20:46.215716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.215147    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-credential-socket" pod "6a0279bf-c6af-4283-9db8-60d0c8dd7a44" (UID: "6a0279bf-c6af-4283-9db8-60d0c8dd7a44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:46.215716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.215314    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-certs" pod "6a0279bf-c6af-4283-9db8-60d0c8dd7a44" (UID: "6a0279bf-c6af-4283-9db8-60d0c8dd7a44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:46.215716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.215630    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.215716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.215654    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.215716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.215671    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.215716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.215685    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.215716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.215700    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.217268 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.217236    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-envoy" pod "6a0279bf-c6af-4283-9db8-60d0c8dd7a44" (UID: "6a0279bf-c6af-4283-9db8-60d0c8dd7a44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:46.217268 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.217260    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-kube-api-access-d9pvm" pod "6a0279bf-c6af-4283-9db8-60d0c8dd7a44" (UID: "6a0279bf-c6af-4283-9db8-60d0c8dd7a44"). InnerVolumeSpecName "kube-api-access-d9pvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:46.217426 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.217275    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-podinfo" pod "6a0279bf-c6af-4283-9db8-60d0c8dd7a44" (UID: "6a0279bf-c6af-4283-9db8-60d0c8dd7a44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:20:46.217426 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.217359    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-token" pod "6a0279bf-c6af-4283-9db8-60d0c8dd7a44" (UID: "6a0279bf-c6af-4283-9db8-60d0c8dd7a44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:46.317476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.317300    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.317476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.317334    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.317476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.317345    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9pvm\" (UniqueName: \"kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-kube-api-access-d9pvm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.317476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.317355    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a0279bf-c6af-4283-9db8-60d0c8dd7a44-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.419637 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.419596    5224 generic.go:356] "Generic (PLEG): container finished" podID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" containerID="1ab9e3d0087e5d676e0a29cd44e0bebd66bd2c42f456d8818c426e3b044b8b14" exitCode=0
Nov 02 20:20:46.419863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.419645    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"
Nov 02 20:20:46.419863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.419666    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" event={"ID":"6a0279bf-c6af-4283-9db8-60d0c8dd7a44","Type":"ContainerDied","Data":"1ab9e3d0087e5d676e0a29cd44e0bebd66bd2c42f456d8818c426e3b044b8b14"}
Nov 02 20:20:46.419863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.419734    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5" event={"ID":"6a0279bf-c6af-4283-9db8-60d0c8dd7a44","Type":"ContainerDied","Data":"97628173c2edaceb0f4749c534fcab3d6d1922831da6633ba4705a6c4ae142d6"}
Nov 02 20:20:46.419863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.419777    5224 scope.go:122] "RemoveContainer" containerID="1ab9e3d0087e5d676e0a29cd44e0bebd66bd2c42f456d8818c426e3b044b8b14"
Nov 02 20:20:46.456795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.453977    5224 scope.go:122] "RemoveContainer" containerID="1ab9e3d0087e5d676e0a29cd44e0bebd66bd2c42f456d8818c426e3b044b8b14"
Nov 02 20:20:46.458406 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:20:46.458366    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab9e3d0087e5d676e0a29cd44e0bebd66bd2c42f456d8818c426e3b044b8b14\": container with ID starting with 1ab9e3d0087e5d676e0a29cd44e0bebd66bd2c42f456d8818c426e3b044b8b14 not found: ID does not exist" containerID="1ab9e3d0087e5d676e0a29cd44e0bebd66bd2c42f456d8818c426e3b044b8b14"
Nov 02 20:20:46.467330 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.467282    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab9e3d0087e5d676e0a29cd44e0bebd66bd2c42f456d8818c426e3b044b8b14"} err="failed to get container status \"1ab9e3d0087e5d676e0a29cd44e0bebd66bd2c42f456d8818c426e3b044b8b14\": rpc error: code = NotFound desc = could not find container \"1ab9e3d0087e5d676e0a29cd44e0bebd66bd2c42f456d8818c426e3b044b8b14\": container with ID starting with 1ab9e3d0087e5d676e0a29cd44e0bebd66bd2c42f456d8818c426e3b044b8b14 not found: ID does not exist"
Nov 02 20:20:46.514326 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:20:46.514291    5224 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0279bf_c6af_4283_9db8_60d0c8dd7a44.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 20:20:46.591053 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.590963    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"]
Nov 02 20:20:46.594558 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:46.594525    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhxw5"]
Nov 02 20:20:47.216795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:47.216417    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" path="/var/lib/kubelet/pods/6a0279bf-c6af-4283-9db8-60d0c8dd7a44/volumes"
Nov 02 20:20:50.351506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:50.351464    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:50.537060 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:50.537021    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:50.713713 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:50.713605    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:55.529792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:55.529717    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:55.699166 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:55.699124    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:56.366934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:20:56.366900    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:20:57.360662 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:20:57.360623    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:21:02.141778 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:02.141729    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:07.038071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:07.038006    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:12.085531 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:12.085476    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:14.010962 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:14.010916    5224 patch_prober.go:29] interesting pod/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:21:14.011479 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:14.011004    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:21:16.633966 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.633875    5224 generic.go:356] "Generic (PLEG): container finished" podID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerID="beae7518d34ac2db33d20bb221a7d303fee816ede3b0432c0a01142b1ca8abbf" exitCode=0
Nov 02 20:21:16.633966 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.633947    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" event={"ID":"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e","Type":"ContainerDied","Data":"beae7518d34ac2db33d20bb221a7d303fee816ede3b0432c0a01142b1ca8abbf"}
Nov 02 20:21:16.634438 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.634420    5224 scope.go:122] "RemoveContainer" containerID="beae7518d34ac2db33d20bb221a7d303fee816ede3b0432c0a01142b1ca8abbf"
Nov 02 20:21:16.636055 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.636027    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:21:16.636127 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.636106    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:21:16.636164 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.636138    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:21:16.648823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.648793    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"]
Nov 02 20:21:16.649349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.649336    5224 cpu_manager.go:414] "containerMap: removing container" podUID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" containerName="istio-proxy"
Nov 02 20:21:16.649420 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.649350    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" containerName="istio-proxy"
Nov 02 20:21:16.649476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.649426    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0279bf-c6af-4283-9db8-60d0c8dd7a44" containerName="istio-proxy"
Nov 02 20:21:16.651926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.651904    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:16.730963 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.730934    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:16.731142 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.731072    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:16.731296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.731275    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-metrics-certs\") pod \"maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:16.731363 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.731323    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4966p\" (UniqueName: \"kubernetes.io/projected/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-kube-api-access-4966p\") pod \"maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:16.739545 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.739510    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:21:16.742476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.742444    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"]
Nov 02 20:21:16.748029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.747996    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3b8db2-serving-cert\""
Nov 02 20:21:16.748029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.748018    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:21:16.748239 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.748045    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 20:21:16.748239 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.748202    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nltm4\""
Nov 02 20:21:16.832871 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.832831    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:16.833094 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.833019    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:16.833159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.833140    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-metrics-certs\") pod \"maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:16.833219 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.833167    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4966p\" (UniqueName: \"kubernetes.io/projected/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-kube-api-access-4966p\") pod \"maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:16.833416 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:21:16.833387    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-serving-cert: secret "maas-api-e2e-ipp-b-3b8db2-serving-cert" not found
Nov 02 20:21:16.833491 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:21:16.833468    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-maas-api-tls podName:b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08 nodeName:}" failed. No retries permitted until 2026-11-02 20:21:17.333443907 +0000 UTC m=+1958.331132511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-maas-api-tls") pod "maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz" (UID: "b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08") : secret "maas-api-e2e-ipp-b-3b8db2-serving-cert" not found
Nov 02 20:21:16.834285 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.834258    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:16.836383 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.836357    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-metrics-certs\") pod \"maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:16.890332 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.890213    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:16.891757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:16.891724    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4966p\" (UniqueName: \"kubernetes.io/projected/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-kube-api-access-4966p\") pod \"maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:17.080269 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:17.080230    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:17.266790 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:17.263462    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:17.340409 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:17.340366    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:17.343133 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:17.343095    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:17.594452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:17.594342    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:17.689417 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:17.689212    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" event={"ID":"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e","Type":"ContainerStarted","Data":"3daa8e804dfccdff746f2c698cf5fd1a94391738d6855daf531650f895a6c20d"}
Nov 02 20:21:18.006436 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:21:18.006389    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36f5cc9_98ed_4eaa_8cd5_cc3b8dd73e08.slice/crio-977dd7821a5f43f5522bbd9d8776e712fd6a93ff9810b075adbff19eca45b424 WatchSource:0}: Error finding container 977dd7821a5f43f5522bbd9d8776e712fd6a93ff9810b075adbff19eca45b424: Status 404 returned error can't find the container with id 977dd7821a5f43f5522bbd9d8776e712fd6a93ff9810b075adbff19eca45b424
Nov 02 20:21:18.009492 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:18.009459    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"]
Nov 02 20:21:18.536779 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:18.536732    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:21:18.536982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:18.536959    5224 patch_prober.go:29] interesting pod/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:21:18.537029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:18.537009    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 20:21:18.697452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:18.697411    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz" event={"ID":"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08","Type":"ContainerStarted","Data":"977dd7821a5f43f5522bbd9d8776e712fd6a93ff9810b075adbff19eca45b424"}
Nov 02 20:21:19.536755 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:19.536713    5224 patch_prober.go:29] interesting pod/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:21:19.536972 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:19.536817    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 20:21:20.536945 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:20.536903    5224 patch_prober.go:29] interesting pod/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:21:20.537416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:20.536977    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 20:21:20.732816 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:20.732761    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:21.082044 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:21.081999    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:21.272795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:21.264393    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:21.537178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:21.537137    5224 patch_prober.go:29] interesting pod/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:21:21.537606 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:21.537197    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 20:21:22.536992 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:22.536950    5224 patch_prober.go:29] interesting pod/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:21:22.537187 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:22.537008    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 20:21:22.683944 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:22.683069    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:23.541912 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:23.541834    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:21:23.542165 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:23.542143    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:21:23.542943 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:23.542922    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:21:23.735077 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:23.735036    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz" event={"ID":"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08","Type":"ContainerStarted","Data":"e6ed8a4c923f5742803b21c5be8f72dcbcd89aa786eb07c43a13c7fd81fe578f"}
Nov 02 20:21:23.735590 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:23.735116    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:23.945414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:23.945299    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz" podStartSLOduration=2.68305359 podStartE2EDuration="7.945282184s" totalImagesPullingTime="5.262228594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:23.944722336 +0000 UTC m=+1964.942410938" watchObservedRunningTime="2026-11-02 20:21:23.945282184 +0000 UTC m=+1964.942970793"
Nov 02 20:21:27.359951 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:21:27.359908    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:21:29.746832 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:29.746796    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:21:29.764009 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:29.763967    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:34.360172 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:34.360106    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:34.700533 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:34.700441    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:21:39.255581 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:39.255523    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:44.190973 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:44.190922    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:48.521038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:48.520949    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:48.955447 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:48.955352    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"]
Nov 02 20:21:49.130058 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:49.130006    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:49.304699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:49.304665    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:49.503207 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:49.503170    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:49.737228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:49.737129    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:53.956313 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:53.956273    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82" podUID="2ccad093-1904-4c78-90e8-d11c2839fe08" containerName="payload-pre-processing" containerID="cri-o://9d1e20028c51f819a7e1be652986fac81313d117463751f98b391151408598a2" gracePeriod=25
Nov 02 20:21:54.229112 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.229085    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"
Nov 02 20:21:54.345133 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.345095    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ccad093-1904-4c78-90e8-d11c2839fe08-kube-api-access-lfspm\" (UniqueName: \"kubernetes.io/projected/2ccad093-1904-4c78-90e8-d11c2839fe08-kube-api-access-lfspm\") pod \"2ccad093-1904-4c78-90e8-d11c2839fe08\" (UID: \"2ccad093-1904-4c78-90e8-d11c2839fe08\") "
Nov 02 20:21:54.345347 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.345189    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ccad093-1904-4c78-90e8-d11c2839fe08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ccad093-1904-4c78-90e8-d11c2839fe08-plugins-config-volume\") pod \"2ccad093-1904-4c78-90e8-d11c2839fe08\" (UID: \"2ccad093-1904-4c78-90e8-d11c2839fe08\") "
Nov 02 20:21:54.345671 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.345626    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ccad093-1904-4c78-90e8-d11c2839fe08-plugins-config-volume" pod "2ccad093-1904-4c78-90e8-d11c2839fe08" (UID: "2ccad093-1904-4c78-90e8-d11c2839fe08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:21:54.346071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.346048    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ccad093-1904-4c78-90e8-d11c2839fe08-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:54.347991 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.347964    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ccad093-1904-4c78-90e8-d11c2839fe08-kube-api-access-lfspm" pod "2ccad093-1904-4c78-90e8-d11c2839fe08" (UID: "2ccad093-1904-4c78-90e8-d11c2839fe08"). InnerVolumeSpecName "kube-api-access-lfspm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:54.447116 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.447078    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfspm\" (UniqueName: \"kubernetes.io/projected/2ccad093-1904-4c78-90e8-d11c2839fe08-kube-api-access-lfspm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:54.928126 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.928082    5224 generic.go:356] "Generic (PLEG): container finished" podID="2ccad093-1904-4c78-90e8-d11c2839fe08" containerID="9d1e20028c51f819a7e1be652986fac81313d117463751f98b391151408598a2" exitCode=0
Nov 02 20:21:54.928329 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.928158    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"
Nov 02 20:21:54.928329 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.928162    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82" event={"ID":"2ccad093-1904-4c78-90e8-d11c2839fe08","Type":"ContainerDied","Data":"9d1e20028c51f819a7e1be652986fac81313d117463751f98b391151408598a2"}
Nov 02 20:21:54.928329 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.928257    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82" event={"ID":"2ccad093-1904-4c78-90e8-d11c2839fe08","Type":"ContainerDied","Data":"cd4ea1958791204f82bb32a7cde1a6853ab3cc86db51dd585f6ab019c8d7eff6"}
Nov 02 20:21:54.928329 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.928284    5224 scope.go:122] "RemoveContainer" containerID="9d1e20028c51f819a7e1be652986fac81313d117463751f98b391151408598a2"
Nov 02 20:21:54.938639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.938619    5224 scope.go:122] "RemoveContainer" containerID="9d1e20028c51f819a7e1be652986fac81313d117463751f98b391151408598a2"
Nov 02 20:21:54.938987 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:21:54.938962    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1e20028c51f819a7e1be652986fac81313d117463751f98b391151408598a2\": container with ID starting with 9d1e20028c51f819a7e1be652986fac81313d117463751f98b391151408598a2 not found: ID does not exist" containerID="9d1e20028c51f819a7e1be652986fac81313d117463751f98b391151408598a2"
Nov 02 20:21:54.939079 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:54.939000    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1e20028c51f819a7e1be652986fac81313d117463751f98b391151408598a2"} err="failed to get container status \"9d1e20028c51f819a7e1be652986fac81313d117463751f98b391151408598a2\": rpc error: code = NotFound desc = could not find container \"9d1e20028c51f819a7e1be652986fac81313d117463751f98b391151408598a2\": container with ID starting with 9d1e20028c51f819a7e1be652986fac81313d117463751f98b391151408598a2 not found: ID does not exist"
Nov 02 20:21:55.095355 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:55.095315    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"]
Nov 02 20:21:55.146573 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:55.146526    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-6fbfc56547-n4k82"]
Nov 02 20:21:55.167889 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:55.167842    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ccad093-1904-4c78-90e8-d11c2839fe08" path="/var/lib/kubelet/pods/2ccad093-1904-4c78-90e8-d11c2839fe08/volumes"
Nov 02 20:21:55.866690 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:55.866657    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:56.273938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:56.273899    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:56.694845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:21:56.694739    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:21:57.359972 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:21:57.359936    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:22:00.139186 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:00.139136    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:00.885439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:00.885402    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:01.145631 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:01.145517    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:01.315234 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:01.315185    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:05.509511 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:05.509468    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:05.800972 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:05.800929    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:06.022307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:06.022261    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:06.515207 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:06.515153    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:08.387180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:08.387118    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:10.238602 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:10.238541    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"]
Nov 02 20:22:10.344874 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:10.344828    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:10.347150 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:10.347126    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"]
Nov 02 20:22:10.566641 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:10.566601    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:10.741576 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:10.741536    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:13.954187 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:13.954132    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:15.241153 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.241077    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj" podUID="23fae0cb-a909-420d-9337-aa80cd97fdb1" containerName="payload-processing" containerID="cri-o://862308ef211fb661894cf2afad36eba471b395244d372a51f5898ba66aebcb03" gracePeriod=25
Nov 02 20:22:15.348305 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.348260    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf" podUID="80860d6f-ec62-427a-a99f-a6b4c4ff3c18" containerName="payload-pre-processing" containerID="cri-o://61a27badd2e7fe7c7ce22e21bace7f7eb5113420a2656ec336b1e73389f017d2" gracePeriod=25
Nov 02 20:22:15.512030 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.511996    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"
Nov 02 20:22:15.612699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.612669    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23fae0cb-a909-420d-9337-aa80cd97fdb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fae0cb-a909-420d-9337-aa80cd97fdb1-plugins-config-volume\") pod \"23fae0cb-a909-420d-9337-aa80cd97fdb1\" (UID: \"23fae0cb-a909-420d-9337-aa80cd97fdb1\") "
Nov 02 20:22:15.612890 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.612841    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fae0cb-a909-420d-9337-aa80cd97fdb1-kube-api-access-6hj56\" (UniqueName: \"kubernetes.io/projected/23fae0cb-a909-420d-9337-aa80cd97fdb1-kube-api-access-6hj56\") pod \"23fae0cb-a909-420d-9337-aa80cd97fdb1\" (UID: \"23fae0cb-a909-420d-9337-aa80cd97fdb1\") "
Nov 02 20:22:15.613132 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.613104    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23fae0cb-a909-420d-9337-aa80cd97fdb1-plugins-config-volume" pod "23fae0cb-a909-420d-9337-aa80cd97fdb1" (UID: "23fae0cb-a909-420d-9337-aa80cd97fdb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:22:15.613299 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.613281    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fae0cb-a909-420d-9337-aa80cd97fdb1-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:22:15.615281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.615256    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fae0cb-a909-420d-9337-aa80cd97fdb1-kube-api-access-6hj56" pod "23fae0cb-a909-420d-9337-aa80cd97fdb1" (UID: "23fae0cb-a909-420d-9337-aa80cd97fdb1"). InnerVolumeSpecName "kube-api-access-6hj56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:22:15.620224 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.620198    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:15.630250 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.630215    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"
Nov 02 20:22:15.714409 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.714366    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-kube-api-access-gpjv9\" (UniqueName: \"kubernetes.io/projected/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-kube-api-access-gpjv9\") pod \"80860d6f-ec62-427a-a99f-a6b4c4ff3c18\" (UID: \"80860d6f-ec62-427a-a99f-a6b4c4ff3c18\") "
Nov 02 20:22:15.714613 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.714436    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-plugins-config-volume\") pod \"80860d6f-ec62-427a-a99f-a6b4c4ff3c18\" (UID: \"80860d6f-ec62-427a-a99f-a6b4c4ff3c18\") "
Nov 02 20:22:15.714928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.714898    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-plugins-config-volume" pod "80860d6f-ec62-427a-a99f-a6b4c4ff3c18" (UID: "80860d6f-ec62-427a-a99f-a6b4c4ff3c18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:22:15.715030 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.714906    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hj56\" (UniqueName: \"kubernetes.io/projected/23fae0cb-a909-420d-9337-aa80cd97fdb1-kube-api-access-6hj56\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:22:15.716605 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.716585    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-kube-api-access-gpjv9" pod "80860d6f-ec62-427a-a99f-a6b4c4ff3c18" (UID: "80860d6f-ec62-427a-a99f-a6b4c4ff3c18"). InnerVolumeSpecName "kube-api-access-gpjv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:22:15.816537 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.816502    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpjv9\" (UniqueName: \"kubernetes.io/projected/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-kube-api-access-gpjv9\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:22:15.816537 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.816536    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80860d6f-ec62-427a-a99f-a6b4c4ff3c18-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:22:15.907354 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:15.907313    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:16.074500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.074405    5224 generic.go:356] "Generic (PLEG): container finished" podID="80860d6f-ec62-427a-a99f-a6b4c4ff3c18" containerID="61a27badd2e7fe7c7ce22e21bace7f7eb5113420a2656ec336b1e73389f017d2" exitCode=0
Nov 02 20:22:16.074500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.074459    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"
Nov 02 20:22:16.074500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.074482    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf" event={"ID":"80860d6f-ec62-427a-a99f-a6b4c4ff3c18","Type":"ContainerDied","Data":"61a27badd2e7fe7c7ce22e21bace7f7eb5113420a2656ec336b1e73389f017d2"}
Nov 02 20:22:16.074804 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.074537    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf" event={"ID":"80860d6f-ec62-427a-a99f-a6b4c4ff3c18","Type":"ContainerDied","Data":"dd35b20fdec16fd344a099eb5f84ced175a82499faacece0201086f549427de7"}
Nov 02 20:22:16.074804 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.074559    5224 scope.go:122] "RemoveContainer" containerID="61a27badd2e7fe7c7ce22e21bace7f7eb5113420a2656ec336b1e73389f017d2"
Nov 02 20:22:16.076187 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.076156    5224 generic.go:356] "Generic (PLEG): container finished" podID="23fae0cb-a909-420d-9337-aa80cd97fdb1" containerID="862308ef211fb661894cf2afad36eba471b395244d372a51f5898ba66aebcb03" exitCode=0
Nov 02 20:22:16.076331 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.076213    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"
Nov 02 20:22:16.076331 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.076316    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj" event={"ID":"23fae0cb-a909-420d-9337-aa80cd97fdb1","Type":"ContainerDied","Data":"862308ef211fb661894cf2afad36eba471b395244d372a51f5898ba66aebcb03"}
Nov 02 20:22:16.076431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.076360    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj" event={"ID":"23fae0cb-a909-420d-9337-aa80cd97fdb1","Type":"ContainerDied","Data":"f209d10ef47d0e75c794081ea48419c1ba83a20b354df8c44b9a46ef0bb89e0f"}
Nov 02 20:22:16.087026 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.086933    5224 scope.go:122] "RemoveContainer" containerID="61a27badd2e7fe7c7ce22e21bace7f7eb5113420a2656ec336b1e73389f017d2"
Nov 02 20:22:16.087383 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:22:16.087357    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a27badd2e7fe7c7ce22e21bace7f7eb5113420a2656ec336b1e73389f017d2\": container with ID starting with 61a27badd2e7fe7c7ce22e21bace7f7eb5113420a2656ec336b1e73389f017d2 not found: ID does not exist" containerID="61a27badd2e7fe7c7ce22e21bace7f7eb5113420a2656ec336b1e73389f017d2"
Nov 02 20:22:16.087491 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.087399    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a27badd2e7fe7c7ce22e21bace7f7eb5113420a2656ec336b1e73389f017d2"} err="failed to get container status \"61a27badd2e7fe7c7ce22e21bace7f7eb5113420a2656ec336b1e73389f017d2\": rpc error: code = NotFound desc = could not find container \"61a27badd2e7fe7c7ce22e21bace7f7eb5113420a2656ec336b1e73389f017d2\": container with ID starting with 61a27badd2e7fe7c7ce22e21bace7f7eb5113420a2656ec336b1e73389f017d2 not found: ID does not exist"
Nov 02 20:22:16.087491 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.087428    5224 scope.go:122] "RemoveContainer" containerID="862308ef211fb661894cf2afad36eba471b395244d372a51f5898ba66aebcb03"
Nov 02 20:22:16.098481 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.098460    5224 scope.go:122] "RemoveContainer" containerID="862308ef211fb661894cf2afad36eba471b395244d372a51f5898ba66aebcb03"
Nov 02 20:22:16.098817 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:22:16.098797    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862308ef211fb661894cf2afad36eba471b395244d372a51f5898ba66aebcb03\": container with ID starting with 862308ef211fb661894cf2afad36eba471b395244d372a51f5898ba66aebcb03 not found: ID does not exist" containerID="862308ef211fb661894cf2afad36eba471b395244d372a51f5898ba66aebcb03"
Nov 02 20:22:16.098870 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.098828    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862308ef211fb661894cf2afad36eba471b395244d372a51f5898ba66aebcb03"} err="failed to get container status \"862308ef211fb661894cf2afad36eba471b395244d372a51f5898ba66aebcb03\": rpc error: code = NotFound desc = could not find container \"862308ef211fb661894cf2afad36eba471b395244d372a51f5898ba66aebcb03\": container with ID starting with 862308ef211fb661894cf2afad36eba471b395244d372a51f5898ba66aebcb03 not found: ID does not exist"
Nov 02 20:22:16.249670 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.249634    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"]
Nov 02 20:22:16.255385 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.255354    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-8949c59-bp2lj"]
Nov 02 20:22:16.366046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.365968    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"]
Nov 02 20:22:16.370074 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:16.370047    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-58449477b7-bxxgf"]
Nov 02 20:22:17.176870 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:17.176828    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23fae0cb-a909-420d-9337-aa80cd97fdb1" path="/var/lib/kubelet/pods/23fae0cb-a909-420d-9337-aa80cd97fdb1/volumes"
Nov 02 20:22:17.177598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:17.177573    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80860d6f-ec62-427a-a99f-a6b4c4ff3c18" path="/var/lib/kubelet/pods/80860d6f-ec62-427a-a99f-a6b4c4ff3c18/volumes"
Nov 02 20:22:18.868690 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:18.868646    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:19.083467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.083413    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:19.203754 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.203667    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"]
Nov 02 20:22:19.204242 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.204227    5224 cpu_manager.go:414] "containerMap: removing container" podUID="23fae0cb-a909-420d-9337-aa80cd97fdb1" containerName="payload-processing"
Nov 02 20:22:19.204287 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.204244    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fae0cb-a909-420d-9337-aa80cd97fdb1" containerName="payload-processing"
Nov 02 20:22:19.204287 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.204259    5224 cpu_manager.go:414] "containerMap: removing container" podUID="80860d6f-ec62-427a-a99f-a6b4c4ff3c18" containerName="payload-pre-processing"
Nov 02 20:22:19.204287 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.204265    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80860d6f-ec62-427a-a99f-a6b4c4ff3c18" containerName="payload-pre-processing"
Nov 02 20:22:19.204384 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.204292    5224 cpu_manager.go:414] "containerMap: removing container" podUID="2ccad093-1904-4c78-90e8-d11c2839fe08" containerName="payload-pre-processing"
Nov 02 20:22:19.204384 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.204297    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ccad093-1904-4c78-90e8-d11c2839fe08" containerName="payload-pre-processing"
Nov 02 20:22:19.204384 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.204371    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ccad093-1904-4c78-90e8-d11c2839fe08" containerName="payload-pre-processing"
Nov 02 20:22:19.204384 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.204384    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fae0cb-a909-420d-9337-aa80cd97fdb1" containerName="payload-processing"
Nov 02 20:22:19.204507 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.204392    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="80860d6f-ec62-427a-a99f-a6b4c4ff3c18" containerName="payload-pre-processing"
Nov 02 20:22:19.209640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.209615    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"
Nov 02 20:22:19.260655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.260619    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"]
Nov 02 20:22:19.332625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.332378    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-408103\""
Nov 02 20:22:19.335304 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.335245    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-408103-dockercfg-2rkgx\""
Nov 02 20:22:19.357251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.356333    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:19.359220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.359184    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e99203c-3219-44e7-9af6-d3de71bbc242-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq\" (UID: \"2e99203c-3219-44e7-9af6-d3de71bbc242\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"
Nov 02 20:22:19.359389 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.359359    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8v6\" (UniqueName: \"kubernetes.io/projected/2e99203c-3219-44e7-9af6-d3de71bbc242-kube-api-access-zx8v6\") pod \"payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq\" (UID: \"2e99203c-3219-44e7-9af6-d3de71bbc242\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"
Nov 02 20:22:19.365557 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.365525    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"]
Nov 02 20:22:19.370510 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.370490    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"
Nov 02 20:22:19.420703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.420667    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"]
Nov 02 20:22:19.462014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.461911    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e99203c-3219-44e7-9af6-d3de71bbc242-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq\" (UID: \"2e99203c-3219-44e7-9af6-d3de71bbc242\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"
Nov 02 20:22:19.462014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.461981    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8deecf9a-e43f-42bb-bc42-6924b0272459-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7\" (UID: \"8deecf9a-e43f-42bb-bc42-6924b0272459\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"
Nov 02 20:22:19.462275 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.462084    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8v6\" (UniqueName: \"kubernetes.io/projected/2e99203c-3219-44e7-9af6-d3de71bbc242-kube-api-access-zx8v6\") pod \"payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq\" (UID: \"2e99203c-3219-44e7-9af6-d3de71bbc242\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"
Nov 02 20:22:19.462275 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.462121    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h9dx\" (UniqueName: \"kubernetes.io/projected/8deecf9a-e43f-42bb-bc42-6924b0272459-kube-api-access-6h9dx\") pod \"payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7\" (UID: \"8deecf9a-e43f-42bb-bc42-6924b0272459\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"
Nov 02 20:22:19.463983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.463304    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e99203c-3219-44e7-9af6-d3de71bbc242-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq\" (UID: \"2e99203c-3219-44e7-9af6-d3de71bbc242\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"
Nov 02 20:22:19.520326 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.520291    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8v6\" (UniqueName: \"kubernetes.io/projected/2e99203c-3219-44e7-9af6-d3de71bbc242-kube-api-access-zx8v6\") pod \"payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq\" (UID: \"2e99203c-3219-44e7-9af6-d3de71bbc242\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"
Nov 02 20:22:19.523150 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.523127    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"
Nov 02 20:22:19.563534 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.563488    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8deecf9a-e43f-42bb-bc42-6924b0272459-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7\" (UID: \"8deecf9a-e43f-42bb-bc42-6924b0272459\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"
Nov 02 20:22:19.563720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.563626    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h9dx\" (UniqueName: \"kubernetes.io/projected/8deecf9a-e43f-42bb-bc42-6924b0272459-kube-api-access-6h9dx\") pod \"payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7\" (UID: \"8deecf9a-e43f-42bb-bc42-6924b0272459\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"
Nov 02 20:22:19.564231 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.564208    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8deecf9a-e43f-42bb-bc42-6924b0272459-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7\" (UID: \"8deecf9a-e43f-42bb-bc42-6924b0272459\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"
Nov 02 20:22:19.622393 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.622355    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h9dx\" (UniqueName: \"kubernetes.io/projected/8deecf9a-e43f-42bb-bc42-6924b0272459-kube-api-access-6h9dx\") pod \"payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7\" (UID: \"8deecf9a-e43f-42bb-bc42-6924b0272459\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"
Nov 02 20:22:19.685256 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.685218    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"
Nov 02 20:22:19.891700 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:22:19.891668    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e99203c_3219_44e7_9af6_d3de71bbc242.slice/crio-860423a9121902c3dc9780091b0306e06efd72080353740f43f184c6cb230b1a WatchSource:0}: Error finding container 860423a9121902c3dc9780091b0306e06efd72080353740f43f184c6cb230b1a: Status 404 returned error can't find the container with id 860423a9121902c3dc9780091b0306e06efd72080353740f43f184c6cb230b1a
Nov 02 20:22:19.895260 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:19.895235    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"]
Nov 02 20:22:20.054863 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:22:20.054826    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8deecf9a_e43f_42bb_bc42_6924b0272459.slice/crio-bfd492be13b44440225ba8c730323024897c3eb6ca3da0737936de876a0573b3 WatchSource:0}: Error finding container bfd492be13b44440225ba8c730323024897c3eb6ca3da0737936de876a0573b3: Status 404 returned error can't find the container with id bfd492be13b44440225ba8c730323024897c3eb6ca3da0737936de876a0573b3
Nov 02 20:22:20.057524 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:20.057500    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"]
Nov 02 20:22:20.117676 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:20.117630    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq" event={"ID":"2e99203c-3219-44e7-9af6-d3de71bbc242","Type":"ContainerStarted","Data":"8cf3fe091f03634418fbe6c3f5f6037839c7869b3ff5958230f7c4c4edebfa5d"}
Nov 02 20:22:20.117821 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:20.117695    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq" event={"ID":"2e99203c-3219-44e7-9af6-d3de71bbc242","Type":"ContainerStarted","Data":"860423a9121902c3dc9780091b0306e06efd72080353740f43f184c6cb230b1a"}
Nov 02 20:22:20.117821 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:20.117724    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"
Nov 02 20:22:20.119245 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:20.119222    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7" event={"ID":"8deecf9a-e43f-42bb-bc42-6924b0272459","Type":"ContainerStarted","Data":"bfd492be13b44440225ba8c730323024897c3eb6ca3da0737936de876a0573b3"}
Nov 02 20:22:20.119364 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:20.119346    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"
Nov 02 20:22:20.234409 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:20.234353    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq" podStartSLOduration=1.234338169 podStartE2EDuration="1.234338169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:20.231820422 +0000 UTC m=+2021.229509030" watchObservedRunningTime="2026-11-02 20:22:20.234338169 +0000 UTC m=+2021.232026777"
Nov 02 20:22:20.348241 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:20.348076    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7" podStartSLOduration=1.348056462 podStartE2EDuration="1.348056462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:20.346271999 +0000 UTC m=+2021.343960620" watchObservedRunningTime="2026-11-02 20:22:20.348056462 +0000 UTC m=+2021.345745071"
Nov 02 20:22:21.127375 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:21.127332    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7" event={"ID":"8deecf9a-e43f-42bb-bc42-6924b0272459","Type":"ContainerStarted","Data":"84cc037f026db9c70da6eee94bef20edbd62dccdd864576dfe559800b3136bfc"}
Nov 02 20:22:24.619780 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:24.619741    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:27.361659 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:22:27.361619    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:22:28.300067 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:28.300028    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:28.821642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:28.821600    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:29.899034 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:29.898998    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"]
Nov 02 20:22:29.903234 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:29.903209    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"
Nov 02 20:22:29.954596 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:29.954558    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"]
Nov 02 20:22:29.980451 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:29.980414    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxm8t\" (UniqueName: \"kubernetes.io/projected/acb5860e-e3ab-4385-a106-84a17f6a0b15-kube-api-access-vxm8t\") pod \"payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9\" (UID: \"acb5860e-e3ab-4385-a106-84a17f6a0b15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"
Nov 02 20:22:29.980642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:29.980480    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb5860e-e3ab-4385-a106-84a17f6a0b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9\" (UID: \"acb5860e-e3ab-4385-a106-84a17f6a0b15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"
Nov 02 20:22:30.003203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.003158    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d3184c\""
Nov 02 20:22:30.003386 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.003260    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d3184c-dockercfg-2shh2\""
Nov 02 20:22:30.082330 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.082294    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxm8t\" (UniqueName: \"kubernetes.io/projected/acb5860e-e3ab-4385-a106-84a17f6a0b15-kube-api-access-vxm8t\") pod \"payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9\" (UID: \"acb5860e-e3ab-4385-a106-84a17f6a0b15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"
Nov 02 20:22:30.082520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.082348    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb5860e-e3ab-4385-a106-84a17f6a0b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9\" (UID: \"acb5860e-e3ab-4385-a106-84a17f6a0b15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"
Nov 02 20:22:30.083134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.083114    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb5860e-e3ab-4385-a106-84a17f6a0b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9\" (UID: \"acb5860e-e3ab-4385-a106-84a17f6a0b15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"
Nov 02 20:22:30.140856 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.140815    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxm8t\" (UniqueName: \"kubernetes.io/projected/acb5860e-e3ab-4385-a106-84a17f6a0b15-kube-api-access-vxm8t\") pod \"payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9\" (UID: \"acb5860e-e3ab-4385-a106-84a17f6a0b15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"
Nov 02 20:22:30.217230 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.217137    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"
Nov 02 20:22:30.586914 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:22:30.586876    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb5860e_e3ab_4385_a106_84a17f6a0b15.slice/crio-ce1014d8754e955cbf6c1c717199835b876b046d0314c615d9322f74b571cef4 WatchSource:0}: Error finding container ce1014d8754e955cbf6c1c717199835b876b046d0314c615d9322f74b571cef4: Status 404 returned error can't find the container with id ce1014d8754e955cbf6c1c717199835b876b046d0314c615d9322f74b571cef4
Nov 02 20:22:30.590863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.590836    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"]
Nov 02 20:22:30.637233 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.637196    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"]
Nov 02 20:22:30.644733 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.644703    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"
Nov 02 20:22:30.690334 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.690289    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a48e6c-6eca-4824-8a08-b612170cc45d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr\" (UID: \"53a48e6c-6eca-4824-8a08-b612170cc45d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"
Nov 02 20:22:30.690536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.690353    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf4lz\" (UniqueName: \"kubernetes.io/projected/53a48e6c-6eca-4824-8a08-b612170cc45d-kube-api-access-pf4lz\") pod \"payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr\" (UID: \"53a48e6c-6eca-4824-8a08-b612170cc45d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"
Nov 02 20:22:30.692861 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.692832    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"]
Nov 02 20:22:30.792208 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.792170    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a48e6c-6eca-4824-8a08-b612170cc45d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr\" (UID: \"53a48e6c-6eca-4824-8a08-b612170cc45d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"
Nov 02 20:22:30.792208 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.792211    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf4lz\" (UniqueName: \"kubernetes.io/projected/53a48e6c-6eca-4824-8a08-b612170cc45d-kube-api-access-pf4lz\") pod \"payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr\" (UID: \"53a48e6c-6eca-4824-8a08-b612170cc45d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"
Nov 02 20:22:30.792918 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.792896    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a48e6c-6eca-4824-8a08-b612170cc45d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr\" (UID: \"53a48e6c-6eca-4824-8a08-b612170cc45d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"
Nov 02 20:22:30.850198 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.850117    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf4lz\" (UniqueName: \"kubernetes.io/projected/53a48e6c-6eca-4824-8a08-b612170cc45d-kube-api-access-pf4lz\") pod \"payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr\" (UID: \"53a48e6c-6eca-4824-8a08-b612170cc45d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"
Nov 02 20:22:30.964027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:30.963980    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"
Nov 02 20:22:31.131678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:31.131596    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"
Nov 02 20:22:31.206513 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:31.206463    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9" event={"ID":"acb5860e-e3ab-4385-a106-84a17f6a0b15","Type":"ContainerStarted","Data":"245ce8c9e3125cd756e3ae8fa3f2961b78e94de66cf566961522e6544800f965"}
Nov 02 20:22:31.206708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:31.206538    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9" event={"ID":"acb5860e-e3ab-4385-a106-84a17f6a0b15","Type":"ContainerStarted","Data":"ce1014d8754e955cbf6c1c717199835b876b046d0314c615d9322f74b571cef4"}
Nov 02 20:22:31.206708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:31.206561    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"
Nov 02 20:22:31.363563 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:31.363493    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9" podStartSLOduration=2.363472445 podStartE2EDuration="2.363472445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:31.362457724 +0000 UTC m=+2032.360146334" watchObservedRunningTime="2026-11-02 20:22:31.363472445 +0000 UTC m=+2032.361161055"
Nov 02 20:22:31.384862 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:22:31.384825    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a48e6c_6eca_4824_8a08_b612170cc45d.slice/crio-4cccfc67e5109c894769b1a8f668c109b6eaa32a04610b579a3744aeee7182d8 WatchSource:0}: Error finding container 4cccfc67e5109c894769b1a8f668c109b6eaa32a04610b579a3744aeee7182d8: Status 404 returned error can't find the container with id 4cccfc67e5109c894769b1a8f668c109b6eaa32a04610b579a3744aeee7182d8
Nov 02 20:22:31.388506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:31.388478    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"]
Nov 02 20:22:31.458876 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:31.458840    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:32.139454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:32.139419    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"
Nov 02 20:22:32.216030 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:32.215992    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr" event={"ID":"53a48e6c-6eca-4824-8a08-b612170cc45d","Type":"ContainerStarted","Data":"0fb912329da68dd55255cc319742c2c0385149ff24b8b38ec7f78f9b38a75130"}
Nov 02 20:22:32.216227 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:32.216057    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr" event={"ID":"53a48e6c-6eca-4824-8a08-b612170cc45d","Type":"ContainerStarted","Data":"4cccfc67e5109c894769b1a8f668c109b6eaa32a04610b579a3744aeee7182d8"}
Nov 02 20:22:32.216227 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:32.216087    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"
Nov 02 20:22:32.372334 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:32.372279    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr" podStartSLOduration=2.372264801 podStartE2EDuration="2.372264801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:32.371672169 +0000 UTC m=+2033.369360759" watchObservedRunningTime="2026-11-02 20:22:32.372264801 +0000 UTC m=+2033.369953410"
Nov 02 20:22:36.567041 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:36.566996    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:41.329433 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:41.329390    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"]
Nov 02 20:22:41.437405 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:41.437352    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"]
Nov 02 20:22:41.669863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:41.669754    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:41.914111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:41.914069    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:42.218994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:42.218957    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"
Nov 02 20:22:43.224577 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:43.224546    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"
Nov 02 20:22:43.568311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:43.568273    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:45.155754 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:45.155709    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:46.330465 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.330402    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm" podUID="e8d32f2b-b566-4c09-803c-8b35ef3aca06" containerName="payload-processing" containerID="cri-o://b824121c944d5a4479d59c29a86c3dfc6d6eb33c0bc6b5658025732c0b588f23" gracePeriod=25
Nov 02 20:22:46.437932 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.437889    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq" podUID="8b53b416-4e46-4fee-9ebc-65f13cfb0f36" containerName="payload-pre-processing" containerID="cri-o://c70a5fc2ba962493d27f262500860eb1b49b7be02195c3d899d008b8dfe509bc" gracePeriod=25
Nov 02 20:22:46.613102 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.613070    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"
Nov 02 20:22:46.708128 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.708100    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8d32f2b-b566-4c09-803c-8b35ef3aca06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d32f2b-b566-4c09-803c-8b35ef3aca06-plugins-config-volume\") pod \"e8d32f2b-b566-4c09-803c-8b35ef3aca06\" (UID: \"e8d32f2b-b566-4c09-803c-8b35ef3aca06\") "
Nov 02 20:22:46.708264 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.708247    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d32f2b-b566-4c09-803c-8b35ef3aca06-kube-api-access-bh52j\" (UniqueName: \"kubernetes.io/projected/e8d32f2b-b566-4c09-803c-8b35ef3aca06-kube-api-access-bh52j\") pod \"e8d32f2b-b566-4c09-803c-8b35ef3aca06\" (UID: \"e8d32f2b-b566-4c09-803c-8b35ef3aca06\") "
Nov 02 20:22:46.708529 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.708504    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d32f2b-b566-4c09-803c-8b35ef3aca06-plugins-config-volume" pod "e8d32f2b-b566-4c09-803c-8b35ef3aca06" (UID: "e8d32f2b-b566-4c09-803c-8b35ef3aca06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:22:46.708839 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.708820    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d32f2b-b566-4c09-803c-8b35ef3aca06-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:22:46.710473 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.710445    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d32f2b-b566-4c09-803c-8b35ef3aca06-kube-api-access-bh52j" pod "e8d32f2b-b566-4c09-803c-8b35ef3aca06" (UID: "e8d32f2b-b566-4c09-803c-8b35ef3aca06"). InnerVolumeSpecName "kube-api-access-bh52j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:22:46.718682 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.718658    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"
Nov 02 20:22:46.809762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.809717    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh52j\" (UniqueName: \"kubernetes.io/projected/e8d32f2b-b566-4c09-803c-8b35ef3aca06-kube-api-access-bh52j\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:22:46.911166 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.911070    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-kube-api-access-wmdqc\" (UniqueName: \"kubernetes.io/projected/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-kube-api-access-wmdqc\") pod \"8b53b416-4e46-4fee-9ebc-65f13cfb0f36\" (UID: \"8b53b416-4e46-4fee-9ebc-65f13cfb0f36\") "
Nov 02 20:22:46.911365 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.911350    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-plugins-config-volume\") pod \"8b53b416-4e46-4fee-9ebc-65f13cfb0f36\" (UID: \"8b53b416-4e46-4fee-9ebc-65f13cfb0f36\") "
Nov 02 20:22:46.911723 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.911697    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-plugins-config-volume" pod "8b53b416-4e46-4fee-9ebc-65f13cfb0f36" (UID: "8b53b416-4e46-4fee-9ebc-65f13cfb0f36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:22:46.913281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:46.913257    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-kube-api-access-wmdqc" pod "8b53b416-4e46-4fee-9ebc-65f13cfb0f36" (UID: "8b53b416-4e46-4fee-9ebc-65f13cfb0f36"). InnerVolumeSpecName "kube-api-access-wmdqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:22:47.012369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.012334    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:22:47.012369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.012366    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmdqc\" (UniqueName: \"kubernetes.io/projected/8b53b416-4e46-4fee-9ebc-65f13cfb0f36-kube-api-access-wmdqc\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:22:47.309028 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.308991    5224 generic.go:356] "Generic (PLEG): container finished" podID="e8d32f2b-b566-4c09-803c-8b35ef3aca06" containerID="b824121c944d5a4479d59c29a86c3dfc6d6eb33c0bc6b5658025732c0b588f23" exitCode=0
Nov 02 20:22:47.309245 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.309051    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"
Nov 02 20:22:47.309245 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.309079    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm" event={"ID":"e8d32f2b-b566-4c09-803c-8b35ef3aca06","Type":"ContainerDied","Data":"b824121c944d5a4479d59c29a86c3dfc6d6eb33c0bc6b5658025732c0b588f23"}
Nov 02 20:22:47.309245 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.309136    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm" event={"ID":"e8d32f2b-b566-4c09-803c-8b35ef3aca06","Type":"ContainerDied","Data":"b8a6869c3fa09299a67bc170fa3844ea60f34c43a338ba26fee78605fa5b1e72"}
Nov 02 20:22:47.309245 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.309161    5224 scope.go:122] "RemoveContainer" containerID="b824121c944d5a4479d59c29a86c3dfc6d6eb33c0bc6b5658025732c0b588f23"
Nov 02 20:22:47.310818 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.310782    5224 generic.go:356] "Generic (PLEG): container finished" podID="8b53b416-4e46-4fee-9ebc-65f13cfb0f36" containerID="c70a5fc2ba962493d27f262500860eb1b49b7be02195c3d899d008b8dfe509bc" exitCode=0
Nov 02 20:22:47.310960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.310834    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq" event={"ID":"8b53b416-4e46-4fee-9ebc-65f13cfb0f36","Type":"ContainerDied","Data":"c70a5fc2ba962493d27f262500860eb1b49b7be02195c3d899d008b8dfe509bc"}
Nov 02 20:22:47.310960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.310860    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"
Nov 02 20:22:47.310960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.310876    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq" event={"ID":"8b53b416-4e46-4fee-9ebc-65f13cfb0f36","Type":"ContainerDied","Data":"bf13db886b0266065480def5cb6f48c8e286e1074893363f3b4b8ebeff196b13"}
Nov 02 20:22:47.322567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.322539    5224 scope.go:122] "RemoveContainer" containerID="b824121c944d5a4479d59c29a86c3dfc6d6eb33c0bc6b5658025732c0b588f23"
Nov 02 20:22:47.322953 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:22:47.322933    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b824121c944d5a4479d59c29a86c3dfc6d6eb33c0bc6b5658025732c0b588f23\": container with ID starting with b824121c944d5a4479d59c29a86c3dfc6d6eb33c0bc6b5658025732c0b588f23 not found: ID does not exist" containerID="b824121c944d5a4479d59c29a86c3dfc6d6eb33c0bc6b5658025732c0b588f23"
Nov 02 20:22:47.323015 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.322965    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b824121c944d5a4479d59c29a86c3dfc6d6eb33c0bc6b5658025732c0b588f23"} err="failed to get container status \"b824121c944d5a4479d59c29a86c3dfc6d6eb33c0bc6b5658025732c0b588f23\": rpc error: code = NotFound desc = could not find container \"b824121c944d5a4479d59c29a86c3dfc6d6eb33c0bc6b5658025732c0b588f23\": container with ID starting with b824121c944d5a4479d59c29a86c3dfc6d6eb33c0bc6b5658025732c0b588f23 not found: ID does not exist"
Nov 02 20:22:47.323015 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.322985    5224 scope.go:122] "RemoveContainer" containerID="c70a5fc2ba962493d27f262500860eb1b49b7be02195c3d899d008b8dfe509bc"
Nov 02 20:22:47.332873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.332679    5224 scope.go:122] "RemoveContainer" containerID="c70a5fc2ba962493d27f262500860eb1b49b7be02195c3d899d008b8dfe509bc"
Nov 02 20:22:47.333146 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:22:47.333028    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70a5fc2ba962493d27f262500860eb1b49b7be02195c3d899d008b8dfe509bc\": container with ID starting with c70a5fc2ba962493d27f262500860eb1b49b7be02195c3d899d008b8dfe509bc not found: ID does not exist" containerID="c70a5fc2ba962493d27f262500860eb1b49b7be02195c3d899d008b8dfe509bc"
Nov 02 20:22:47.333146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.333059    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70a5fc2ba962493d27f262500860eb1b49b7be02195c3d899d008b8dfe509bc"} err="failed to get container status \"c70a5fc2ba962493d27f262500860eb1b49b7be02195c3d899d008b8dfe509bc\": rpc error: code = NotFound desc = could not find container \"c70a5fc2ba962493d27f262500860eb1b49b7be02195c3d899d008b8dfe509bc\": container with ID starting with c70a5fc2ba962493d27f262500860eb1b49b7be02195c3d899d008b8dfe509bc not found: ID does not exist"
Nov 02 20:22:47.476025 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.475918    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"]
Nov 02 20:22:47.479012 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.478980    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-9bfbb46dc-jb6zq"]
Nov 02 20:22:47.638154 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.638115    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"]
Nov 02 20:22:47.640551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:47.640529    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-7786fccb45-mbpdm"]
Nov 02 20:22:48.384874 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:48.384830    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:49.169869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:49.169822    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b53b416-4e46-4fee-9ebc-65f13cfb0f36" path="/var/lib/kubelet/pods/8b53b416-4e46-4fee-9ebc-65f13cfb0f36/volumes"
Nov 02 20:22:49.170370 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:49.170348    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8d32f2b-b566-4c09-803c-8b35ef3aca06" path="/var/lib/kubelet/pods/e8d32f2b-b566-4c09-803c-8b35ef3aca06/volumes"
Nov 02 20:22:50.878204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:50.878165    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:53.691040 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:53.690994    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:56.144234 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:56.144192    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:57.012682 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:57.012647    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:57.360721 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:22:57.360619    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:22:57.682428 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:57.682312    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:22:58.278369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:22:58.278326    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:01.098715 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:01.098679    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:06.163071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:06.163029    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:12.689350 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:12.689306    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:17.785347 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:17.785307    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:18.618975 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:18.618934    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:21.424018 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:21.423979    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:21.910430 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:21.910390    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:22.928822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:22.928786    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"]
Nov 02 20:23:22.929364 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:22.929352    5224 cpu_manager.go:414] "containerMap: removing container" podUID="8b53b416-4e46-4fee-9ebc-65f13cfb0f36" containerName="payload-pre-processing"
Nov 02 20:23:22.929427 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:22.929365    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b53b416-4e46-4fee-9ebc-65f13cfb0f36" containerName="payload-pre-processing"
Nov 02 20:23:22.929427 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:22.929382    5224 cpu_manager.go:414] "containerMap: removing container" podUID="e8d32f2b-b566-4c09-803c-8b35ef3aca06" containerName="payload-processing"
Nov 02 20:23:22.929427 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:22.929390    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d32f2b-b566-4c09-803c-8b35ef3aca06" containerName="payload-processing"
Nov 02 20:23:22.929559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:22.929480    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b53b416-4e46-4fee-9ebc-65f13cfb0f36" containerName="payload-pre-processing"
Nov 02 20:23:22.929559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:22.929489    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d32f2b-b566-4c09-803c-8b35ef3aca06" containerName="payload-processing"
Nov 02 20:23:22.933588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:22.933564    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"
Nov 02 20:23:22.984803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:22.984733    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"]
Nov 02 20:23:23.033349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.033315    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e87e9a\""
Nov 02 20:23:23.033349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.033338    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e87e9a-dockercfg-2vp8t\""
Nov 02 20:23:23.047069 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.047032    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fc8090-e113-4fdc-a497-20d55f78b354-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj\" (UID: \"44fc8090-e113-4fdc-a497-20d55f78b354\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"
Nov 02 20:23:23.047260 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.047089    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgqrz\" (UniqueName: \"kubernetes.io/projected/44fc8090-e113-4fdc-a497-20d55f78b354-kube-api-access-vgqrz\") pod \"payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj\" (UID: \"44fc8090-e113-4fdc-a497-20d55f78b354\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"
Nov 02 20:23:23.148646 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.148606    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fc8090-e113-4fdc-a497-20d55f78b354-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj\" (UID: \"44fc8090-e113-4fdc-a497-20d55f78b354\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"
Nov 02 20:23:23.148869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.148661    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgqrz\" (UniqueName: \"kubernetes.io/projected/44fc8090-e113-4fdc-a497-20d55f78b354-kube-api-access-vgqrz\") pod \"payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj\" (UID: \"44fc8090-e113-4fdc-a497-20d55f78b354\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"
Nov 02 20:23:23.149404 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.149378    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fc8090-e113-4fdc-a497-20d55f78b354-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj\" (UID: \"44fc8090-e113-4fdc-a497-20d55f78b354\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"
Nov 02 20:23:23.206614 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.206539    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgqrz\" (UniqueName: \"kubernetes.io/projected/44fc8090-e113-4fdc-a497-20d55f78b354-kube-api-access-vgqrz\") pod \"payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj\" (UID: \"44fc8090-e113-4fdc-a497-20d55f78b354\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"
Nov 02 20:23:23.247961 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.247928    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"
Nov 02 20:23:23.617841 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:23:23.617804    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fc8090_e113_4fdc_a497_20d55f78b354.slice/crio-832aea3729bd1d5264f77843e1512a44813ed28808918e8f61b4622eea09cfd2 WatchSource:0}: Error finding container 832aea3729bd1d5264f77843e1512a44813ed28808918e8f61b4622eea09cfd2: Status 404 returned error can't find the container with id 832aea3729bd1d5264f77843e1512a44813ed28808918e8f61b4622eea09cfd2
Nov 02 20:23:23.620623 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.620592    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"]
Nov 02 20:23:23.712281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.712242    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"]
Nov 02 20:23:23.716335 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.716315    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"
Nov 02 20:23:23.755162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.755125    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7035711a-f7cb-48f5-a7fc-d66673ea55c2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj\" (UID: \"7035711a-f7cb-48f5-a7fc-d66673ea55c2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"
Nov 02 20:23:23.755360 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.755341    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhj8j\" (UniqueName: \"kubernetes.io/projected/7035711a-f7cb-48f5-a7fc-d66673ea55c2-kube-api-access-dhj8j\") pod \"payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj\" (UID: \"7035711a-f7cb-48f5-a7fc-d66673ea55c2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"
Nov 02 20:23:23.768183 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.768151    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"]
Nov 02 20:23:23.856431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.856390    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhj8j\" (UniqueName: \"kubernetes.io/projected/7035711a-f7cb-48f5-a7fc-d66673ea55c2-kube-api-access-dhj8j\") pod \"payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj\" (UID: \"7035711a-f7cb-48f5-a7fc-d66673ea55c2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"
Nov 02 20:23:23.856622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.856494    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7035711a-f7cb-48f5-a7fc-d66673ea55c2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj\" (UID: \"7035711a-f7cb-48f5-a7fc-d66673ea55c2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"
Nov 02 20:23:23.857342 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.857314    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7035711a-f7cb-48f5-a7fc-d66673ea55c2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj\" (UID: \"7035711a-f7cb-48f5-a7fc-d66673ea55c2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"
Nov 02 20:23:23.914532 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:23.914463    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhj8j\" (UniqueName: \"kubernetes.io/projected/7035711a-f7cb-48f5-a7fc-d66673ea55c2-kube-api-access-dhj8j\") pod \"payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj\" (UID: \"7035711a-f7cb-48f5-a7fc-d66673ea55c2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"
Nov 02 20:23:24.031493 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:24.031448    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"
Nov 02 20:23:24.406491 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:23:24.406442    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7035711a_f7cb_48f5_a7fc_d66673ea55c2.slice/crio-5f9f04156077cde8ce3140f3c901ffeb377e56933a19abeba13cd7c546ae76d2 WatchSource:0}: Error finding container 5f9f04156077cde8ce3140f3c901ffeb377e56933a19abeba13cd7c546ae76d2: Status 404 returned error can't find the container with id 5f9f04156077cde8ce3140f3c901ffeb377e56933a19abeba13cd7c546ae76d2
Nov 02 20:23:24.409355 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:24.409330    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"]
Nov 02 20:23:24.485566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:24.485517    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:24.558073 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:24.558035    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj" event={"ID":"44fc8090-e113-4fdc-a497-20d55f78b354","Type":"ContainerStarted","Data":"4f954375cc4f8711b84351b41b2eeb394078d04491d1a11c1afc5fed5a1e9375"}
Nov 02 20:23:24.558285 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:24.558100    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj" event={"ID":"44fc8090-e113-4fdc-a497-20d55f78b354","Type":"ContainerStarted","Data":"832aea3729bd1d5264f77843e1512a44813ed28808918e8f61b4622eea09cfd2"}
Nov 02 20:23:24.558285 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:24.558126    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"
Nov 02 20:23:24.560362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:24.560328    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj" event={"ID":"7035711a-f7cb-48f5-a7fc-d66673ea55c2","Type":"ContainerStarted","Data":"399bcee2f496afe7f643f5614da41832972ac594c8f0796dc63fec67d7000896"}
Nov 02 20:23:24.560603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:24.560387    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj" event={"ID":"7035711a-f7cb-48f5-a7fc-d66673ea55c2","Type":"ContainerStarted","Data":"5f9f04156077cde8ce3140f3c901ffeb377e56933a19abeba13cd7c546ae76d2"}
Nov 02 20:23:24.561136 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:24.561110    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"
Nov 02 20:23:24.675461 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:24.675352    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj" podStartSLOduration=2.675336992 podStartE2EDuration="2.675336992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:24.674889751 +0000 UTC m=+2085.672578361" watchObservedRunningTime="2026-11-02 20:23:24.675336992 +0000 UTC m=+2085.673025633"
Nov 02 20:23:24.705047 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:24.705000    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:24.793532 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:24.793459    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj" podStartSLOduration=1.7934351560000001 podStartE2EDuration="1.793435156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:24.791064389 +0000 UTC m=+2085.788753001" watchObservedRunningTime="2026-11-02 20:23:24.793435156 +0000 UTC m=+2085.791123766"
Nov 02 20:23:24.902953 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:24.902911    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:27.361594 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:23:27.361549    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:23:29.388641 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:29.388602    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:29.934637 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:29.934598    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"]
Nov 02 20:23:29.934937 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:29.934895    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz" podUID="b5fd73b1-9a09-4db1-8d06-c56bd893a471" containerName="payload-pre-processing" containerID="cri-o://0e45d2d87877ff2d8c2c5804080d8c0a16057237d3b2f3c323bca035d97de704" gracePeriod=30
Nov 02 20:23:30.109371 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.109333    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:30.220073 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.220043    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"
Nov 02 20:23:30.290044 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.290007    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:30.343058 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.343017    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5fd73b1-9a09-4db1-8d06-c56bd893a471-kube-api-access-t6mf4\" (UniqueName: \"kubernetes.io/projected/b5fd73b1-9a09-4db1-8d06-c56bd893a471-kube-api-access-t6mf4\") pod \"b5fd73b1-9a09-4db1-8d06-c56bd893a471\" (UID: \"b5fd73b1-9a09-4db1-8d06-c56bd893a471\") "
Nov 02 20:23:30.343260 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.343103    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5fd73b1-9a09-4db1-8d06-c56bd893a471-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fd73b1-9a09-4db1-8d06-c56bd893a471-plugins-config-volume\") pod \"b5fd73b1-9a09-4db1-8d06-c56bd893a471\" (UID: \"b5fd73b1-9a09-4db1-8d06-c56bd893a471\") "
Nov 02 20:23:30.343506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.343469    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fd73b1-9a09-4db1-8d06-c56bd893a471-plugins-config-volume" pod "b5fd73b1-9a09-4db1-8d06-c56bd893a471" (UID: "b5fd73b1-9a09-4db1-8d06-c56bd893a471"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:23:30.343687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.343668    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fd73b1-9a09-4db1-8d06-c56bd893a471-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:23:30.345820 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.345763    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fd73b1-9a09-4db1-8d06-c56bd893a471-kube-api-access-t6mf4" pod "b5fd73b1-9a09-4db1-8d06-c56bd893a471" (UID: "b5fd73b1-9a09-4db1-8d06-c56bd893a471"). InnerVolumeSpecName "kube-api-access-t6mf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:23:30.449868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.446273    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6mf4\" (UniqueName: \"kubernetes.io/projected/b5fd73b1-9a09-4db1-8d06-c56bd893a471-kube-api-access-t6mf4\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:23:30.603369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.603332    5224 generic.go:356] "Generic (PLEG): container finished" podID="b5fd73b1-9a09-4db1-8d06-c56bd893a471" containerID="0e45d2d87877ff2d8c2c5804080d8c0a16057237d3b2f3c323bca035d97de704" exitCode=0
Nov 02 20:23:30.603588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.603389    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"
Nov 02 20:23:30.603588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.603421    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz" event={"ID":"b5fd73b1-9a09-4db1-8d06-c56bd893a471","Type":"ContainerDied","Data":"0e45d2d87877ff2d8c2c5804080d8c0a16057237d3b2f3c323bca035d97de704"}
Nov 02 20:23:30.603588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.603485    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz" event={"ID":"b5fd73b1-9a09-4db1-8d06-c56bd893a471","Type":"ContainerDied","Data":"18e0a5f3f95fee319ae382eaa6940e1c93f9bee3209b8b91c6cbc773d648239d"}
Nov 02 20:23:30.603588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.603517    5224 scope.go:122] "RemoveContainer" containerID="0e45d2d87877ff2d8c2c5804080d8c0a16057237d3b2f3c323bca035d97de704"
Nov 02 20:23:30.615119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.615011    5224 scope.go:122] "RemoveContainer" containerID="0e45d2d87877ff2d8c2c5804080d8c0a16057237d3b2f3c323bca035d97de704"
Nov 02 20:23:30.618004 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:23:30.617972    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e45d2d87877ff2d8c2c5804080d8c0a16057237d3b2f3c323bca035d97de704\": container with ID starting with 0e45d2d87877ff2d8c2c5804080d8c0a16057237d3b2f3c323bca035d97de704 not found: ID does not exist" containerID="0e45d2d87877ff2d8c2c5804080d8c0a16057237d3b2f3c323bca035d97de704"
Nov 02 20:23:30.618108 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.618015    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e45d2d87877ff2d8c2c5804080d8c0a16057237d3b2f3c323bca035d97de704"} err="failed to get container status \"0e45d2d87877ff2d8c2c5804080d8c0a16057237d3b2f3c323bca035d97de704\": rpc error: code = NotFound desc = could not find container \"0e45d2d87877ff2d8c2c5804080d8c0a16057237d3b2f3c323bca035d97de704\": container with ID starting with 0e45d2d87877ff2d8c2c5804080d8c0a16057237d3b2f3c323bca035d97de704 not found: ID does not exist"
Nov 02 20:23:30.779590 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.771949    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"]
Nov 02 20:23:30.779590 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:30.778729    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-dcfc9c699-nrnnz"]
Nov 02 20:23:31.079073 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.078968    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:31.167881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.167846    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5fd73b1-9a09-4db1-8d06-c56bd893a471" path="/var/lib/kubelet/pods/b5fd73b1-9a09-4db1-8d06-c56bd893a471/volumes"
Nov 02 20:23:31.198626 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.198591    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"]
Nov 02 20:23:31.199176 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.199159    5224 cpu_manager.go:414] "containerMap: removing container" podUID="b5fd73b1-9a09-4db1-8d06-c56bd893a471" containerName="payload-pre-processing"
Nov 02 20:23:31.199237 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.199177    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5fd73b1-9a09-4db1-8d06-c56bd893a471" containerName="payload-pre-processing"
Nov 02 20:23:31.199295 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.199285    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5fd73b1-9a09-4db1-8d06-c56bd893a471" containerName="payload-pre-processing"
Nov 02 20:23:31.202735 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.202717    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.256266 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.256233    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-408103-serving-cert\""
Nov 02 20:23:31.257581 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.257537    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"]
Nov 02 20:23:31.258205 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.258167    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:31.358167 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.358067    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2003aed-acbb-4307-9733-fd6e0b791217-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-408103-565c587697-rcntg\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.358339 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.358190    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqlx\" (UniqueName: \"kubernetes.io/projected/b2003aed-acbb-4307-9733-fd6e0b791217-kube-api-access-vsqlx\") pod \"maas-api-e2e-ipp-b-408103-565c587697-rcntg\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.358339 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.358242    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-maas-api-tls\") pod \"maas-api-e2e-ipp-b-408103-565c587697-rcntg\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.358430 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.358345    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-metrics-certs\") pod \"maas-api-e2e-ipp-b-408103-565c587697-rcntg\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.451305 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.451265    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:31.460170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.460088    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2003aed-acbb-4307-9733-fd6e0b791217-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-408103-565c587697-rcntg\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.462175 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.462052    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2003aed-acbb-4307-9733-fd6e0b791217-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-408103-565c587697-rcntg\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.462412 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.462270    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqlx\" (UniqueName: \"kubernetes.io/projected/b2003aed-acbb-4307-9733-fd6e0b791217-kube-api-access-vsqlx\") pod \"maas-api-e2e-ipp-b-408103-565c587697-rcntg\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.462504 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.462482    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-maas-api-tls\") pod \"maas-api-e2e-ipp-b-408103-565c587697-rcntg\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.462625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.462606    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-metrics-certs\") pod \"maas-api-e2e-ipp-b-408103-565c587697-rcntg\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.467557 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.467524    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-metrics-certs\") pod \"maas-api-e2e-ipp-b-408103-565c587697-rcntg\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.469312 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.469286    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-maas-api-tls\") pod \"maas-api-e2e-ipp-b-408103-565c587697-rcntg\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.521080 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.521045    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqlx\" (UniqueName: \"kubernetes.io/projected/b2003aed-acbb-4307-9733-fd6e0b791217-kube-api-access-vsqlx\") pod \"maas-api-e2e-ipp-b-408103-565c587697-rcntg\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.644274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.644175    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:31.817628 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.817592    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:31.841598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:31.841559    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:32.189199 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:23:32.189165    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2003aed_acbb_4307_9733_fd6e0b791217.slice/crio-38e6ca7899ae4879568a534f4f7e5b1db1b19b46e0167ecbeca54d2b898597ed WatchSource:0}: Error finding container 38e6ca7899ae4879568a534f4f7e5b1db1b19b46e0167ecbeca54d2b898597ed: Status 404 returned error can't find the container with id 38e6ca7899ae4879568a534f4f7e5b1db1b19b46e0167ecbeca54d2b898597ed
Nov 02 20:23:32.191916 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:32.191892    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"]
Nov 02 20:23:32.619904 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:32.619867    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg" event={"ID":"b2003aed-acbb-4307-9733-fd6e0b791217","Type":"ContainerStarted","Data":"38e6ca7899ae4879568a534f4f7e5b1db1b19b46e0167ecbeca54d2b898597ed"}
Nov 02 20:23:33.022778 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:33.022498    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:33.523727 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:33.523686    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:33.649520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:33.649455    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg" event={"ID":"b2003aed-acbb-4307-9733-fd6e0b791217","Type":"ContainerStarted","Data":"e8d7b222b150a62debccd7cc341d16d161a4c4a363fd07e91832e4b95b0b98b8"}
Nov 02 20:23:33.651887 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:33.650556    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:33.767973 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:33.767917    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg" podStartSLOduration=1.936378687 podStartE2EDuration="2.767901724s" totalImagesPullingTime="831.523037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:33.767561858 +0000 UTC m=+2094.765250467" watchObservedRunningTime="2026-11-02 20:23:33.767901724 +0000 UTC m=+2094.765590368"
Nov 02 20:23:34.314596 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:34.314558    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:35.569896 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:35.569867    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"
Nov 02 20:23:36.577763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:36.577731    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"
Nov 02 20:23:36.846354 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:36.846248    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:40.104530 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:40.104490    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:40.669888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:40.669858    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:23:42.119718 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:42.119682    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:43.108065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:43.108027    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:23:46.537396 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:46.537356    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:46.879900 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:46.879791    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"]
Nov 02 20:23:46.880134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:46.880086    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr" podUID="53a48e6c-6eca-4824-8a08-b612170cc45d" containerName="payload-processing" containerID="cri-o://0fb912329da68dd55255cc319742c2c0385149ff24b8b38ec7f78f9b38a75130" gracePeriod=30
Nov 02 20:23:46.989530 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:46.989482    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"]
Nov 02 20:23:46.989835 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:46.989780    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9" podUID="acb5860e-e3ab-4385-a106-84a17f6a0b15" containerName="payload-pre-processing" containerID="cri-o://245ce8c9e3125cd756e3ae8fa3f2961b78e94de66cf566961522e6544800f965" gracePeriod=30
Nov 02 20:23:47.081347 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.081306    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:47.169532 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.169497    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"
Nov 02 20:23:47.256449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.256393    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:47.269475 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.269448    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a48e6c-6eca-4824-8a08-b612170cc45d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a48e6c-6eca-4824-8a08-b612170cc45d-plugins-config-volume\") pod \"53a48e6c-6eca-4824-8a08-b612170cc45d\" (UID: \"53a48e6c-6eca-4824-8a08-b612170cc45d\") "
Nov 02 20:23:47.269620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.269587    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a48e6c-6eca-4824-8a08-b612170cc45d-kube-api-access-pf4lz\" (UniqueName: \"kubernetes.io/projected/53a48e6c-6eca-4824-8a08-b612170cc45d-kube-api-access-pf4lz\") pod \"53a48e6c-6eca-4824-8a08-b612170cc45d\" (UID: \"53a48e6c-6eca-4824-8a08-b612170cc45d\") "
Nov 02 20:23:47.269926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.269899    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a48e6c-6eca-4824-8a08-b612170cc45d-plugins-config-volume" pod "53a48e6c-6eca-4824-8a08-b612170cc45d" (UID: "53a48e6c-6eca-4824-8a08-b612170cc45d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:23:47.270275 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.270254    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a48e6c-6eca-4824-8a08-b612170cc45d-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:23:47.271803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.271752    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a48e6c-6eca-4824-8a08-b612170cc45d-kube-api-access-pf4lz" pod "53a48e6c-6eca-4824-8a08-b612170cc45d" (UID: "53a48e6c-6eca-4824-8a08-b612170cc45d"). InnerVolumeSpecName "kube-api-access-pf4lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:23:47.277231 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.277210    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"
Nov 02 20:23:47.370750 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.370708    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acb5860e-e3ab-4385-a106-84a17f6a0b15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb5860e-e3ab-4385-a106-84a17f6a0b15-plugins-config-volume\") pod \"acb5860e-e3ab-4385-a106-84a17f6a0b15\" (UID: \"acb5860e-e3ab-4385-a106-84a17f6a0b15\") "
Nov 02 20:23:47.371016 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.370882    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb5860e-e3ab-4385-a106-84a17f6a0b15-kube-api-access-vxm8t\" (UniqueName: \"kubernetes.io/projected/acb5860e-e3ab-4385-a106-84a17f6a0b15-kube-api-access-vxm8t\") pod \"acb5860e-e3ab-4385-a106-84a17f6a0b15\" (UID: \"acb5860e-e3ab-4385-a106-84a17f6a0b15\") "
Nov 02 20:23:47.371218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.371202    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf4lz\" (UniqueName: \"kubernetes.io/projected/53a48e6c-6eca-4824-8a08-b612170cc45d-kube-api-access-pf4lz\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:23:47.371277 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.371210    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb5860e-e3ab-4385-a106-84a17f6a0b15-plugins-config-volume" pod "acb5860e-e3ab-4385-a106-84a17f6a0b15" (UID: "acb5860e-e3ab-4385-a106-84a17f6a0b15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:23:47.372995 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.372975    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb5860e-e3ab-4385-a106-84a17f6a0b15-kube-api-access-vxm8t" pod "acb5860e-e3ab-4385-a106-84a17f6a0b15" (UID: "acb5860e-e3ab-4385-a106-84a17f6a0b15"). InnerVolumeSpecName "kube-api-access-vxm8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:23:47.472850 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.472378    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb5860e-e3ab-4385-a106-84a17f6a0b15-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:23:47.472850 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.472412    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxm8t\" (UniqueName: \"kubernetes.io/projected/acb5860e-e3ab-4385-a106-84a17f6a0b15-kube-api-access-vxm8t\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:23:47.738242 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.738140    5224 generic.go:356] "Generic (PLEG): container finished" podID="acb5860e-e3ab-4385-a106-84a17f6a0b15" containerID="245ce8c9e3125cd756e3ae8fa3f2961b78e94de66cf566961522e6544800f965" exitCode=0
Nov 02 20:23:47.738242 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.738201    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"
Nov 02 20:23:47.738242 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.738215    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9" event={"ID":"acb5860e-e3ab-4385-a106-84a17f6a0b15","Type":"ContainerDied","Data":"245ce8c9e3125cd756e3ae8fa3f2961b78e94de66cf566961522e6544800f965"}
Nov 02 20:23:47.738902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.738275    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9" event={"ID":"acb5860e-e3ab-4385-a106-84a17f6a0b15","Type":"ContainerDied","Data":"ce1014d8754e955cbf6c1c717199835b876b046d0314c615d9322f74b571cef4"}
Nov 02 20:23:47.738902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.738306    5224 scope.go:122] "RemoveContainer" containerID="245ce8c9e3125cd756e3ae8fa3f2961b78e94de66cf566961522e6544800f965"
Nov 02 20:23:47.739710 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.739686    5224 generic.go:356] "Generic (PLEG): container finished" podID="53a48e6c-6eca-4824-8a08-b612170cc45d" containerID="0fb912329da68dd55255cc319742c2c0385149ff24b8b38ec7f78f9b38a75130" exitCode=0
Nov 02 20:23:47.739875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.739736    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr" event={"ID":"53a48e6c-6eca-4824-8a08-b612170cc45d","Type":"ContainerDied","Data":"0fb912329da68dd55255cc319742c2c0385149ff24b8b38ec7f78f9b38a75130"}
Nov 02 20:23:47.739875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.739743    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"
Nov 02 20:23:47.739875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.739786    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr" event={"ID":"53a48e6c-6eca-4824-8a08-b612170cc45d","Type":"ContainerDied","Data":"4cccfc67e5109c894769b1a8f668c109b6eaa32a04610b579a3744aeee7182d8"}
Nov 02 20:23:47.757141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.752037    5224 scope.go:122] "RemoveContainer" containerID="245ce8c9e3125cd756e3ae8fa3f2961b78e94de66cf566961522e6544800f965"
Nov 02 20:23:47.757141 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:23:47.752524    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245ce8c9e3125cd756e3ae8fa3f2961b78e94de66cf566961522e6544800f965\": container with ID starting with 245ce8c9e3125cd756e3ae8fa3f2961b78e94de66cf566961522e6544800f965 not found: ID does not exist" containerID="245ce8c9e3125cd756e3ae8fa3f2961b78e94de66cf566961522e6544800f965"
Nov 02 20:23:47.757141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.752569    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245ce8c9e3125cd756e3ae8fa3f2961b78e94de66cf566961522e6544800f965"} err="failed to get container status \"245ce8c9e3125cd756e3ae8fa3f2961b78e94de66cf566961522e6544800f965\": rpc error: code = NotFound desc = could not find container \"245ce8c9e3125cd756e3ae8fa3f2961b78e94de66cf566961522e6544800f965\": container with ID starting with 245ce8c9e3125cd756e3ae8fa3f2961b78e94de66cf566961522e6544800f965 not found: ID does not exist"
Nov 02 20:23:47.757141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.752597    5224 scope.go:122] "RemoveContainer" containerID="0fb912329da68dd55255cc319742c2c0385149ff24b8b38ec7f78f9b38a75130"
Nov 02 20:23:47.765061 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.765037    5224 scope.go:122] "RemoveContainer" containerID="0fb912329da68dd55255cc319742c2c0385149ff24b8b38ec7f78f9b38a75130"
Nov 02 20:23:47.765380 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:23:47.765362    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb912329da68dd55255cc319742c2c0385149ff24b8b38ec7f78f9b38a75130\": container with ID starting with 0fb912329da68dd55255cc319742c2c0385149ff24b8b38ec7f78f9b38a75130 not found: ID does not exist" containerID="0fb912329da68dd55255cc319742c2c0385149ff24b8b38ec7f78f9b38a75130"
Nov 02 20:23:47.765418 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.765392    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb912329da68dd55255cc319742c2c0385149ff24b8b38ec7f78f9b38a75130"} err="failed to get container status \"0fb912329da68dd55255cc319742c2c0385149ff24b8b38ec7f78f9b38a75130\": rpc error: code = NotFound desc = could not find container \"0fb912329da68dd55255cc319742c2c0385149ff24b8b38ec7f78f9b38a75130\": container with ID starting with 0fb912329da68dd55255cc319742c2c0385149ff24b8b38ec7f78f9b38a75130 not found: ID does not exist"
Nov 02 20:23:47.905566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.905537    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"]
Nov 02 20:23:47.908089 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:47.908065    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-5bf74c4d75-5xhw9"]
Nov 02 20:23:48.017912 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:48.017879    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"]
Nov 02 20:23:48.021459 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:48.021426    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3184c-f58555d58-hs7zr"]
Nov 02 20:23:48.634074 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:48.634035    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:49.170380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:49.170343    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a48e6c-6eca-4824-8a08-b612170cc45d" path="/var/lib/kubelet/pods/53a48e6c-6eca-4824-8a08-b612170cc45d/volumes"
Nov 02 20:23:49.170865 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:49.170848    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acb5860e-e3ab-4385-a106-84a17f6a0b15" path="/var/lib/kubelet/pods/acb5860e-e3ab-4385-a106-84a17f6a0b15/volumes"
Nov 02 20:23:52.228446 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:52.228406    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:52.753543 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:52.753506    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:54.322179 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:54.322139    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:54.498015 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:54.497974    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:54.831504 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:54.831460    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:55.229895 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.229798    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"]
Nov 02 20:23:55.230083 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.230007    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl" podUID="9eb40531-747d-4ee6-99d3-cf76163f4b83" containerName="payload-processing" containerID="cri-o://9122be1fd785b0dd7bef145df58280d834647b37b0c5025d23dad2b0976c1100" gracePeriod=30
Nov 02 20:23:55.438813 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.436525    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:55.502725 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.502697    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"
Nov 02 20:23:55.586408 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.586367    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb40531-747d-4ee6-99d3-cf76163f4b83-kube-api-access-cmbpr\" (UniqueName: \"kubernetes.io/projected/9eb40531-747d-4ee6-99d3-cf76163f4b83-kube-api-access-cmbpr\") pod \"9eb40531-747d-4ee6-99d3-cf76163f4b83\" (UID: \"9eb40531-747d-4ee6-99d3-cf76163f4b83\") "
Nov 02 20:23:55.586603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.586436    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb40531-747d-4ee6-99d3-cf76163f4b83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb40531-747d-4ee6-99d3-cf76163f4b83-plugins-config-volume\") pod \"9eb40531-747d-4ee6-99d3-cf76163f4b83\" (UID: \"9eb40531-747d-4ee6-99d3-cf76163f4b83\") "
Nov 02 20:23:55.586869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.586834    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb40531-747d-4ee6-99d3-cf76163f4b83-plugins-config-volume" pod "9eb40531-747d-4ee6-99d3-cf76163f4b83" (UID: "9eb40531-747d-4ee6-99d3-cf76163f4b83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:23:55.588577 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.588556    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb40531-747d-4ee6-99d3-cf76163f4b83-kube-api-access-cmbpr" pod "9eb40531-747d-4ee6-99d3-cf76163f4b83" (UID: "9eb40531-747d-4ee6-99d3-cf76163f4b83"). InnerVolumeSpecName "kube-api-access-cmbpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:23:55.661742 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.661703    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:55.687818 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.687784    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmbpr\" (UniqueName: \"kubernetes.io/projected/9eb40531-747d-4ee6-99d3-cf76163f4b83-kube-api-access-cmbpr\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:23:55.687818 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.687815    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb40531-747d-4ee6-99d3-cf76163f4b83-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:23:55.800131 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.800095    5224 generic.go:356] "Generic (PLEG): container finished" podID="9eb40531-747d-4ee6-99d3-cf76163f4b83" containerID="9122be1fd785b0dd7bef145df58280d834647b37b0c5025d23dad2b0976c1100" exitCode=0
Nov 02 20:23:55.800346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.800148    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"
Nov 02 20:23:55.800346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.800169    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl" event={"ID":"9eb40531-747d-4ee6-99d3-cf76163f4b83","Type":"ContainerDied","Data":"9122be1fd785b0dd7bef145df58280d834647b37b0c5025d23dad2b0976c1100"}
Nov 02 20:23:55.800346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.800224    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl" event={"ID":"9eb40531-747d-4ee6-99d3-cf76163f4b83","Type":"ContainerDied","Data":"576f3681973a3af7dab4a58955d0d0f6f60df792349b1187f6246b287894610f"}
Nov 02 20:23:55.800346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.800251    5224 scope.go:122] "RemoveContainer" containerID="9122be1fd785b0dd7bef145df58280d834647b37b0c5025d23dad2b0976c1100"
Nov 02 20:23:55.810953 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.810936    5224 scope.go:122] "RemoveContainer" containerID="9122be1fd785b0dd7bef145df58280d834647b37b0c5025d23dad2b0976c1100"
Nov 02 20:23:55.811251 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:23:55.811231    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9122be1fd785b0dd7bef145df58280d834647b37b0c5025d23dad2b0976c1100\": container with ID starting with 9122be1fd785b0dd7bef145df58280d834647b37b0c5025d23dad2b0976c1100 not found: ID does not exist" containerID="9122be1fd785b0dd7bef145df58280d834647b37b0c5025d23dad2b0976c1100"
Nov 02 20:23:55.811296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.811260    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9122be1fd785b0dd7bef145df58280d834647b37b0c5025d23dad2b0976c1100"} err="failed to get container status \"9122be1fd785b0dd7bef145df58280d834647b37b0c5025d23dad2b0976c1100\": rpc error: code = NotFound desc = could not find container \"9122be1fd785b0dd7bef145df58280d834647b37b0c5025d23dad2b0976c1100\": container with ID starting with 9122be1fd785b0dd7bef145df58280d834647b37b0c5025d23dad2b0976c1100 not found: ID does not exist"
Nov 02 20:23:55.872819 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.872750    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:55.994747 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:55.994674    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"]
Nov 02 20:23:56.014788 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:56.004378    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-296ae9-57bfcdcf4b-h2bkl"]
Nov 02 20:23:56.392472 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:56.392435    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:56.565022 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:56.564985    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:56.899707 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:56.899669    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:57.137890 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:57.137850    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:57.177913 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:57.177823    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb40531-747d-4ee6-99d3-cf76163f4b83" path="/var/lib/kubelet/pods/9eb40531-747d-4ee6-99d3-cf76163f4b83/volumes"
Nov 02 20:23:57.308582 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:57.308535    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:57.361504 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:23:57.361469    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:23:57.484362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:57.484250    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:58.022422 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:58.022369    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:58.655729 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:58.655684    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:58.853364 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:58.853305    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:59.130680 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:59.130630    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:59.307319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:59.307280    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:23:59.899547 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:23:59.899491    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:02.063378 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:02.063332    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:02.809615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:02.809576    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:03.041198 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:03.041160    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:05.700681 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:05.700636    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:05.990543 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:05.990450    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:06.158551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:06.158508    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:06.973158 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:06.973122    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:07.476516 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:07.476479    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:07.691146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:07.691109    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:08.984051 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:08.984011    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:11.186496 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:11.186458    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:11.362585 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:11.362545    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:11.545490 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:11.545451    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:12.104321 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:12.104286    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:12.563004 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:12.562964    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:24:12.819267 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:12.819164    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:13.268142 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:13.268099    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:14.659462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:14.659420    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:14.832024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:14.831974    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:15.366605 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:15.366543    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:15.823570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:15.823525    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:17.569362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:17.569325    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:20.000493 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:20.000460    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:20.195903 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:20.195858    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:20.414549 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:20.414505    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:22.036660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:22.036612    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:23.171822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:23.171782    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:23.342603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:23.342555    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:27.360227 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:24:27.360181    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:24:27.606757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:27.606707    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:34.777632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:34.777591    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:39.660259 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:39.660215    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:44.506641 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:44.506588    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:44.890506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:44.890471    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:45.328239 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:45.328198    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:45.950687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:45.949196    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:46.522591 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:46.522008    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:46.705793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:46.700587    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:46.893029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:46.892924    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:48.017497 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:48.017453    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:48.654140 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:48.654106    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"]
Nov 02 20:24:48.761969 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:48.761932    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"]
Nov 02 20:24:49.034909 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:49.034863    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:49.202932 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:49.202895    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:50.261542 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:50.261493    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:51.108859 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:51.108817    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:51.361585 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:51.361491    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:52.555661 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.555621    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"]
Nov 02 20:24:52.556221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.556205    5224 cpu_manager.go:414] "containerMap: removing container" podUID="9eb40531-747d-4ee6-99d3-cf76163f4b83" containerName="payload-processing"
Nov 02 20:24:52.556267 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.556222    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb40531-747d-4ee6-99d3-cf76163f4b83" containerName="payload-processing"
Nov 02 20:24:52.556267 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.556246    5224 cpu_manager.go:414] "containerMap: removing container" podUID="acb5860e-e3ab-4385-a106-84a17f6a0b15" containerName="payload-pre-processing"
Nov 02 20:24:52.556267 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.556255    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb5860e-e3ab-4385-a106-84a17f6a0b15" containerName="payload-pre-processing"
Nov 02 20:24:52.556382 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.556272    5224 cpu_manager.go:414] "containerMap: removing container" podUID="53a48e6c-6eca-4824-8a08-b612170cc45d" containerName="payload-processing"
Nov 02 20:24:52.556382 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.556280    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a48e6c-6eca-4824-8a08-b612170cc45d" containerName="payload-processing"
Nov 02 20:24:52.556467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.556383    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb40531-747d-4ee6-99d3-cf76163f4b83" containerName="payload-processing"
Nov 02 20:24:52.556467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.556396    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb5860e-e3ab-4385-a106-84a17f6a0b15" containerName="payload-pre-processing"
Nov 02 20:24:52.556467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.556416    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a48e6c-6eca-4824-8a08-b612170cc45d" containerName="payload-processing"
Nov 02 20:24:52.561423 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.561403    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.612270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.612236    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"]
Nov 02 20:24:52.655712 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.655677    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f01ab5-ac30-406c-b8a6-63a76e042836-tls-certs\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.655936 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.655738    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-tmp-dir\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.655936 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.655883    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-dshm\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.656049 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.655979    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-model-cache\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.656049 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.656012    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-home\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.656158 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.656083    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rhwh\" (UniqueName: \"kubernetes.io/projected/65f01ab5-ac30-406c-b8a6-63a76e042836-kube-api-access-9rhwh\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.660457 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.660428    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e87e9a\"/\"openshift-service-ca.crt\""
Nov 02 20:24:52.660457 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.660428    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e87e9a\"/\"kube-root-ca.crt\""
Nov 02 20:24:52.660648 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.660496    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e87e9a\"/\"default-dockercfg-twnpk\""
Nov 02 20:24:52.666644 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.666620    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e87e9a\"/\"ipp-route-e87e9a-kserve-self-signed-certs\""
Nov 02 20:24:52.695062 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.695026    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:52.757823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.757781    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-dshm\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.758025 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.757867    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-model-cache\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.758025 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.757895    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-home\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.758025 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.757931    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rhwh\" (UniqueName: \"kubernetes.io/projected/65f01ab5-ac30-406c-b8a6-63a76e042836-kube-api-access-9rhwh\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.758025 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.758001    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f01ab5-ac30-406c-b8a6-63a76e042836-tls-certs\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.758257 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.758049    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-tmp-dir\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.758373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.758343    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-model-cache\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.758445 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.758371    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-home\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.758445 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.758431    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-tmp-dir\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.760589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.760547    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-dshm\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.760713 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.760692    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f01ab5-ac30-406c-b8a6-63a76e042836-tls-certs\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.817878 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.817792    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rhwh\" (UniqueName: \"kubernetes.io/projected/65f01ab5-ac30-406c-b8a6-63a76e042836-kube-api-access-9rhwh\") pod \"ipp-route-e87e9a-kserve-76788fb5bf-lnnr8\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") " pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:52.874869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:52.874827    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:53.255562 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:24:53.255523    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65f01ab5_ac30_406c_b8a6_63a76e042836.slice/crio-90d4f6b7cab3a0325aaff0d1af7fa1d56144e5d7ec8c48421559158aa58ea6ae WatchSource:0}: Error finding container 90d4f6b7cab3a0325aaff0d1af7fa1d56144e5d7ec8c48421559158aa58ea6ae: Status 404 returned error can't find the container with id 90d4f6b7cab3a0325aaff0d1af7fa1d56144e5d7ec8c48421559158aa58ea6ae
Nov 02 20:24:53.258915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:53.258894    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"]
Nov 02 20:24:53.654754 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:53.654701    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7" podUID="8deecf9a-e43f-42bb-bc42-6924b0272459" containerName="payload-processing" containerID="cri-o://84cc037f026db9c70da6eee94bef20edbd62dccdd864576dfe559800b3136bfc" gracePeriod=25
Nov 02 20:24:53.762885 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:53.762837    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq" podUID="2e99203c-3219-44e7-9af6-d3de71bbc242" containerName="payload-pre-processing" containerID="cri-o://8cf3fe091f03634418fbe6c3f5f6037839c7869b3ff5958230f7c4c4edebfa5d" gracePeriod=25
Nov 02 20:24:53.954638 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:53.954608    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"
Nov 02 20:24:54.057223 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.057194    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"
Nov 02 20:24:54.076242 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.076208    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8deecf9a-e43f-42bb-bc42-6924b0272459-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8deecf9a-e43f-42bb-bc42-6924b0272459-plugins-config-volume\") pod \"8deecf9a-e43f-42bb-bc42-6924b0272459\" (UID: \"8deecf9a-e43f-42bb-bc42-6924b0272459\") "
Nov 02 20:24:54.076447 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.076363    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8deecf9a-e43f-42bb-bc42-6924b0272459-kube-api-access-6h9dx\" (UniqueName: \"kubernetes.io/projected/8deecf9a-e43f-42bb-bc42-6924b0272459-kube-api-access-6h9dx\") pod \"8deecf9a-e43f-42bb-bc42-6924b0272459\" (UID: \"8deecf9a-e43f-42bb-bc42-6924b0272459\") "
Nov 02 20:24:54.076688 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.076659    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8deecf9a-e43f-42bb-bc42-6924b0272459-plugins-config-volume" pod "8deecf9a-e43f-42bb-bc42-6924b0272459" (UID: "8deecf9a-e43f-42bb-bc42-6924b0272459"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:24:54.077193 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.076835    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8deecf9a-e43f-42bb-bc42-6924b0272459-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:24:54.079368 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.079342    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8deecf9a-e43f-42bb-bc42-6924b0272459-kube-api-access-6h9dx" pod "8deecf9a-e43f-42bb-bc42-6924b0272459" (UID: "8deecf9a-e43f-42bb-bc42-6924b0272459"). InnerVolumeSpecName "kube-api-access-6h9dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:24:54.177877 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.177783    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e99203c-3219-44e7-9af6-d3de71bbc242-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e99203c-3219-44e7-9af6-d3de71bbc242-plugins-config-volume\") pod \"2e99203c-3219-44e7-9af6-d3de71bbc242\" (UID: \"2e99203c-3219-44e7-9af6-d3de71bbc242\") "
Nov 02 20:24:54.178060 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.177997    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e99203c-3219-44e7-9af6-d3de71bbc242-kube-api-access-zx8v6\" (UniqueName: \"kubernetes.io/projected/2e99203c-3219-44e7-9af6-d3de71bbc242-kube-api-access-zx8v6\") pod \"2e99203c-3219-44e7-9af6-d3de71bbc242\" (UID: \"2e99203c-3219-44e7-9af6-d3de71bbc242\") "
Nov 02 20:24:54.178257 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.178230    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e99203c-3219-44e7-9af6-d3de71bbc242-plugins-config-volume" pod "2e99203c-3219-44e7-9af6-d3de71bbc242" (UID: "2e99203c-3219-44e7-9af6-d3de71bbc242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:24:54.178568 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.178548    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h9dx\" (UniqueName: \"kubernetes.io/projected/8deecf9a-e43f-42bb-bc42-6924b0272459-kube-api-access-6h9dx\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:24:54.178657 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.178571    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e99203c-3219-44e7-9af6-d3de71bbc242-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:24:54.180476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.180447    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e99203c-3219-44e7-9af6-d3de71bbc242-kube-api-access-zx8v6" pod "2e99203c-3219-44e7-9af6-d3de71bbc242" (UID: "2e99203c-3219-44e7-9af6-d3de71bbc242"). InnerVolumeSpecName "kube-api-access-zx8v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:24:54.191644 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.191611    5224 generic.go:356] "Generic (PLEG): container finished" podID="2e99203c-3219-44e7-9af6-d3de71bbc242" containerID="8cf3fe091f03634418fbe6c3f5f6037839c7869b3ff5958230f7c4c4edebfa5d" exitCode=0
Nov 02 20:24:54.191840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.191671    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"
Nov 02 20:24:54.191840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.191703    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq" event={"ID":"2e99203c-3219-44e7-9af6-d3de71bbc242","Type":"ContainerDied","Data":"8cf3fe091f03634418fbe6c3f5f6037839c7869b3ff5958230f7c4c4edebfa5d"}
Nov 02 20:24:54.191840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.191802    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq" event={"ID":"2e99203c-3219-44e7-9af6-d3de71bbc242","Type":"ContainerDied","Data":"860423a9121902c3dc9780091b0306e06efd72080353740f43f184c6cb230b1a"}
Nov 02 20:24:54.191840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.191836    5224 scope.go:122] "RemoveContainer" containerID="8cf3fe091f03634418fbe6c3f5f6037839c7869b3ff5958230f7c4c4edebfa5d"
Nov 02 20:24:54.194208 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.194185    5224 generic.go:356] "Generic (PLEG): container finished" podID="8deecf9a-e43f-42bb-bc42-6924b0272459" containerID="84cc037f026db9c70da6eee94bef20edbd62dccdd864576dfe559800b3136bfc" exitCode=0
Nov 02 20:24:54.194328 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.194241    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"
Nov 02 20:24:54.194328 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.194279    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7" event={"ID":"8deecf9a-e43f-42bb-bc42-6924b0272459","Type":"ContainerDied","Data":"84cc037f026db9c70da6eee94bef20edbd62dccdd864576dfe559800b3136bfc"}
Nov 02 20:24:54.194464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.194346    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7" event={"ID":"8deecf9a-e43f-42bb-bc42-6924b0272459","Type":"ContainerDied","Data":"bfd492be13b44440225ba8c730323024897c3eb6ca3da0737936de876a0573b3"}
Nov 02 20:24:54.204079 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.203586    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8" event={"ID":"65f01ab5-ac30-406c-b8a6-63a76e042836","Type":"ContainerStarted","Data":"ea259808605e1261f071baee6771c5eb987019f138c623850208711f24c97159"}
Nov 02 20:24:54.204079 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.203668    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8" event={"ID":"65f01ab5-ac30-406c-b8a6-63a76e042836","Type":"ContainerStarted","Data":"90d4f6b7cab3a0325aaff0d1af7fa1d56144e5d7ec8c48421559158aa58ea6ae"}
Nov 02 20:24:54.204079 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.203970    5224 scope.go:122] "RemoveContainer" containerID="8cf3fe091f03634418fbe6c3f5f6037839c7869b3ff5958230f7c4c4edebfa5d"
Nov 02 20:24:54.204463 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:24:54.204414    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf3fe091f03634418fbe6c3f5f6037839c7869b3ff5958230f7c4c4edebfa5d\": container with ID starting with 8cf3fe091f03634418fbe6c3f5f6037839c7869b3ff5958230f7c4c4edebfa5d not found: ID does not exist" containerID="8cf3fe091f03634418fbe6c3f5f6037839c7869b3ff5958230f7c4c4edebfa5d"
Nov 02 20:24:54.204686 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.204621    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf3fe091f03634418fbe6c3f5f6037839c7869b3ff5958230f7c4c4edebfa5d"} err="failed to get container status \"8cf3fe091f03634418fbe6c3f5f6037839c7869b3ff5958230f7c4c4edebfa5d\": rpc error: code = NotFound desc = could not find container \"8cf3fe091f03634418fbe6c3f5f6037839c7869b3ff5958230f7c4c4edebfa5d\": container with ID starting with 8cf3fe091f03634418fbe6c3f5f6037839c7869b3ff5958230f7c4c4edebfa5d not found: ID does not exist"
Nov 02 20:24:54.204686 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.204658    5224 scope.go:122] "RemoveContainer" containerID="84cc037f026db9c70da6eee94bef20edbd62dccdd864576dfe559800b3136bfc"
Nov 02 20:24:54.217607 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.217578    5224 scope.go:122] "RemoveContainer" containerID="84cc037f026db9c70da6eee94bef20edbd62dccdd864576dfe559800b3136bfc"
Nov 02 20:24:54.218062 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:24:54.218038    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84cc037f026db9c70da6eee94bef20edbd62dccdd864576dfe559800b3136bfc\": container with ID starting with 84cc037f026db9c70da6eee94bef20edbd62dccdd864576dfe559800b3136bfc not found: ID does not exist" containerID="84cc037f026db9c70da6eee94bef20edbd62dccdd864576dfe559800b3136bfc"
Nov 02 20:24:54.218165 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.218073    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84cc037f026db9c70da6eee94bef20edbd62dccdd864576dfe559800b3136bfc"} err="failed to get container status \"84cc037f026db9c70da6eee94bef20edbd62dccdd864576dfe559800b3136bfc\": rpc error: code = NotFound desc = could not find container \"84cc037f026db9c70da6eee94bef20edbd62dccdd864576dfe559800b3136bfc\": container with ID starting with 84cc037f026db9c70da6eee94bef20edbd62dccdd864576dfe559800b3136bfc not found: ID does not exist"
Nov 02 20:24:54.250849 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.250812    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:54.280314 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.280280    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx8v6\" (UniqueName: \"kubernetes.io/projected/2e99203c-3219-44e7-9af6-d3de71bbc242-kube-api-access-zx8v6\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:24:54.368126 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.368073    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8" podStartSLOduration=2.368057488 podStartE2EDuration="2.368057488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:54.365595249 +0000 UTC m=+2175.363283858" watchObservedRunningTime="2026-11-02 20:24:54.368057488 +0000 UTC m=+2175.365746097"
Nov 02 20:24:54.479601 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.479497    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:54.481999 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.481969    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"]
Nov 02 20:24:54.486246 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.486213    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-59bbddbc6c-g6grq"]
Nov 02 20:24:54.597707 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.597668    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"]
Nov 02 20:24:54.602389 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.602357    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-408103-57c7cff599-h2fn7"]
Nov 02 20:24:54.874958 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:54.874925    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:24:55.169671 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:55.169594    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e99203c-3219-44e7-9af6-d3de71bbc242" path="/var/lib/kubelet/pods/2e99203c-3219-44e7-9af6-d3de71bbc242/volumes"
Nov 02 20:24:55.170180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:55.170164    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8deecf9a-e43f-42bb-bc42-6924b0272459" path="/var/lib/kubelet/pods/8deecf9a-e43f-42bb-bc42-6924b0272459/volumes"
Nov 02 20:24:55.837749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:55.837701    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:55.887515 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:55.887472    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:24:56.989211 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:56.989167    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:24:57.359820 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:24:57.359778    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:24:57.719544 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:24:57.719440    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:02.852274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:02.852218    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:02.875928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:02.875892    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:25:02.891104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:02.891074    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:25:03.273288 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:03.273259    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:25:07.582857 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:07.582812    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:07.798628 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:07.798582    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:07.975683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:07.975570    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:12.668528 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:12.668462    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:16.669253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:16.669215    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"]
Nov 02 20:25:16.669677 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:16.669498    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x" podUID="ed0d9cbd-fa51-44ea-8333-2b7db2337e89" containerName="istio-proxy" containerID="cri-o://438d750cb13d9a2da07d5ba1de8a784af52522123f05ce288feede5238971ec0" gracePeriod=30
Nov 02 20:25:17.562455 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:17.562412    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:25:17.562648 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:17.562492    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x" podUID="ed0d9cbd-fa51-44ea-8333-2b7db2337e89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:25:17.570441 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:17.570380    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:21.819186 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:21.819130    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:21.980092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:21.980052    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:25:22.132864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.132816    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-certs\") pod \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") "
Nov 02 20:25:22.133090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.132917    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istiod-ca-cert\") pod \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") "
Nov 02 20:25:22.133090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.132982    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-kube-api-access-5ndkh\" (UniqueName: \"kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-kube-api-access-5ndkh\") pod \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") "
Nov 02 20:25:22.133090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.133031    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-envoy\") pod \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") "
Nov 02 20:25:22.133269 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.133110    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-token\" (UniqueName: \"kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-token\") pod \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") "
Nov 02 20:25:22.133269 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.133151    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-podinfo\") pod \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") "
Nov 02 20:25:22.133269 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.133186    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-certs" pod "ed0d9cbd-fa51-44ea-8333-2b7db2337e89" (UID: "ed0d9cbd-fa51-44ea-8333-2b7db2337e89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:25:22.133426 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.133277    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-credential-socket\") pod \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") "
Nov 02 20:25:22.133488 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.133428    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istiod-ca-cert" pod "ed0d9cbd-fa51-44ea-8333-2b7db2337e89" (UID: "ed0d9cbd-fa51-44ea-8333-2b7db2337e89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:25:22.134097 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.134068    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-credential-socket" pod "ed0d9cbd-fa51-44ea-8333-2b7db2337e89" (UID: "ed0d9cbd-fa51-44ea-8333-2b7db2337e89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:25:22.134238 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.134166    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-socket\") pod \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") "
Nov 02 20:25:22.134238 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.134198    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-data\") pod \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\" (UID: \"ed0d9cbd-fa51-44ea-8333-2b7db2337e89\") "
Nov 02 20:25:22.134848 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.134823    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:22.134848 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.134849    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:22.135046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.134864    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:22.135104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.135070    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-data" pod "ed0d9cbd-fa51-44ea-8333-2b7db2337e89" (UID: "ed0d9cbd-fa51-44ea-8333-2b7db2337e89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:25:22.135262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.135243    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-socket" pod "ed0d9cbd-fa51-44ea-8333-2b7db2337e89" (UID: "ed0d9cbd-fa51-44ea-8333-2b7db2337e89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:25:22.136114 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.136087    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-podinfo" pod "ed0d9cbd-fa51-44ea-8333-2b7db2337e89" (UID: "ed0d9cbd-fa51-44ea-8333-2b7db2337e89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:25:22.136228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.136202    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-envoy" pod "ed0d9cbd-fa51-44ea-8333-2b7db2337e89" (UID: "ed0d9cbd-fa51-44ea-8333-2b7db2337e89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:25:22.136284 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.136209    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-token" pod "ed0d9cbd-fa51-44ea-8333-2b7db2337e89" (UID: "ed0d9cbd-fa51-44ea-8333-2b7db2337e89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:25:22.137583 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.137546    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-kube-api-access-5ndkh" pod "ed0d9cbd-fa51-44ea-8333-2b7db2337e89" (UID: "ed0d9cbd-fa51-44ea-8333-2b7db2337e89"). InnerVolumeSpecName "kube-api-access-5ndkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:25:22.236235 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.236137    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:22.236235 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.236175    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:22.236235 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.236186    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ndkh\" (UniqueName: \"kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-kube-api-access-5ndkh\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:22.236235 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.236198    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:22.236235 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.236210    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:22.236235 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.236220    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0d9cbd-fa51-44ea-8333-2b7db2337e89-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:22.347939 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.347891    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:22.381454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.381416    5224 generic.go:356] "Generic (PLEG): container finished" podID="ed0d9cbd-fa51-44ea-8333-2b7db2337e89" containerID="438d750cb13d9a2da07d5ba1de8a784af52522123f05ce288feede5238971ec0" exitCode=0
Nov 02 20:25:22.381675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.381478    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"
Nov 02 20:25:22.381675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.381500    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x" event={"ID":"ed0d9cbd-fa51-44ea-8333-2b7db2337e89","Type":"ContainerDied","Data":"438d750cb13d9a2da07d5ba1de8a784af52522123f05ce288feede5238971ec0"}
Nov 02 20:25:22.381675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.381564    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x" event={"ID":"ed0d9cbd-fa51-44ea-8333-2b7db2337e89","Type":"ContainerDied","Data":"c9449c2de5c9136960f85a7d05fa8c7c5ac2a74671821b0e2a815a135448807f"}
Nov 02 20:25:22.381675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.381596    5224 scope.go:122] "RemoveContainer" containerID="438d750cb13d9a2da07d5ba1de8a784af52522123f05ce288feede5238971ec0"
Nov 02 20:25:22.392862 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.392837    5224 scope.go:122] "RemoveContainer" containerID="438d750cb13d9a2da07d5ba1de8a784af52522123f05ce288feede5238971ec0"
Nov 02 20:25:22.393229 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:25:22.393208    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438d750cb13d9a2da07d5ba1de8a784af52522123f05ce288feede5238971ec0\": container with ID starting with 438d750cb13d9a2da07d5ba1de8a784af52522123f05ce288feede5238971ec0 not found: ID does not exist" containerID="438d750cb13d9a2da07d5ba1de8a784af52522123f05ce288feede5238971ec0"
Nov 02 20:25:22.393329 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.393234    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438d750cb13d9a2da07d5ba1de8a784af52522123f05ce288feede5238971ec0"} err="failed to get container status \"438d750cb13d9a2da07d5ba1de8a784af52522123f05ce288feede5238971ec0\": rpc error: code = NotFound desc = could not find container \"438d750cb13d9a2da07d5ba1de8a784af52522123f05ce288feede5238971ec0\": container with ID starting with 438d750cb13d9a2da07d5ba1de8a784af52522123f05ce288feede5238971ec0 not found: ID does not exist"
Nov 02 20:25:22.515751 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.515705    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:22.549499 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.549456    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"]
Nov 02 20:25:22.554053 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:22.554023    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-zdn7x"]
Nov 02 20:25:23.168352 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:23.168307    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed0d9cbd-fa51-44ea-8333-2b7db2337e89" path="/var/lib/kubelet/pods/ed0d9cbd-fa51-44ea-8333-2b7db2337e89/volumes"
Nov 02 20:25:23.384132 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:23.384096    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:23.565021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:23.564984    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:23.877000 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:23.876895    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:24.049466 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:24.049426    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:25.380194 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:25.380141    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:25.923947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:25.923901    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:26.673341 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:26.673302    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:27.362422 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:25:27.362387    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:25:29.188120 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:29.188077    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:31.632231 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:31.632189    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:31.810155 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:31.810109    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:32.394482 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:32.394440    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:34.764324 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:34.764281    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:41.738483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:41.738443    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:45.454401 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:45.454356    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:45.926837 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:45.926763    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:48.969996 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:48.969960    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:50.582674 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:50.582630    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:25:52.596030 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:52.595984    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:53.089940 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:53.089893    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:55.998263 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:55.998218    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:57.258483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:57.258445    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:25:57.360478 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:25:57.360443    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:25:58.042266 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:25:58.042221    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:00.822108 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:00.822059    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:01.003251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:01.003180    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:01.179650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:01.179535    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:05.969629 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:05.969591    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:10.560349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.560304    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"]
Nov 02 20:26:10.560928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.560590    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz" podUID="b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08" containerName="maas-api" containerID="cri-o://e6ed8a4c923f5742803b21c5be8f72dcbcd89aa786eb07c43a13c7fd81fe578f" gracePeriod=30
Nov 02 20:26:10.683807 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.683754    5224 generic.go:356] "Generic (PLEG): container finished" podID="b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08" containerID="e6ed8a4c923f5742803b21c5be8f72dcbcd89aa786eb07c43a13c7fd81fe578f" exitCode=0
Nov 02 20:26:10.683942 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.683805    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz" event={"ID":"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08","Type":"ContainerDied","Data":"e6ed8a4c923f5742803b21c5be8f72dcbcd89aa786eb07c43a13c7fd81fe578f"}
Nov 02 20:26:10.704461 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.704421    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:10.866306 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.866270    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:26:10.949102 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.949047    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-openshift-service-ca\") pod \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") "
Nov 02 20:26:10.949541 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.949506    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-metrics-certs\") pod \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") "
Nov 02 20:26:10.949711 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.949557    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-maas-api-tls\") pod \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") "
Nov 02 20:26:10.949711 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.949621    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-kube-api-access-4966p\" (UniqueName: \"kubernetes.io/projected/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-kube-api-access-4966p\") pod \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\" (UID: \"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08\") "
Nov 02 20:26:10.953640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.953603    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-openshift-service-ca" pod "b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08" (UID: "b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:26:10.954556 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.954505    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-kube-api-access-4966p" pod "b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08" (UID: "b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08"). InnerVolumeSpecName "kube-api-access-4966p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:26:10.956781 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.956715    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-metrics-certs" pod "b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08" (UID: "b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:26:10.957845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:10.957810    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-maas-api-tls" pod "b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08" (UID: "b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:26:11.051994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:11.051948    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:26:11.051994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:11.051986    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:26:11.051994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:11.052000    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:26:11.052264 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:11.052013    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4966p\" (UniqueName: \"kubernetes.io/projected/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08-kube-api-access-4966p\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:26:11.212940 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:11.212906    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:11.384726 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:11.384680    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:11.692020 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:11.691980    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz" event={"ID":"b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08","Type":"ContainerDied","Data":"977dd7821a5f43f5522bbd9d8776e712fd6a93ff9810b075adbff19eca45b424"}
Nov 02 20:26:11.692020 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:11.692009    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"
Nov 02 20:26:11.692545 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:11.692048    5224 scope.go:122] "RemoveContainer" containerID="e6ed8a4c923f5742803b21c5be8f72dcbcd89aa786eb07c43a13c7fd81fe578f"
Nov 02 20:26:11.859422 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:11.859379    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"]
Nov 02 20:26:11.910567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:11.910527    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b8db2-f5ff4696b-kdfgz"]
Nov 02 20:26:12.579014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:12.578978    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:12.753230 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:12.753189    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:13.168436 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:13.168402    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08" path="/var/lib/kubelet/pods/b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08/volumes"
Nov 02 20:26:14.200537 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:14.200496    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:14.677756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:14.677710    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:16.226318 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:16.226279    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:16.914247 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:16.914201    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:17.205555 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:17.205446    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:17.910708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:17.910668    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:20.652949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:20.652897    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:20.874204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:20.874164    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:21.157939 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:21.157898    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:23.202683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:23.202638    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:25.906640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:25.906597    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:25.995614 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:25.995572    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:26:26.159804 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:26.159679    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:26.337186 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:26.337149    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:27.113384 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:27.113345    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:27.363447 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:26:27.363344    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:26:28.102403 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:28.102369    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:28.272723 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:28.272687    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:28.759312 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:28.759268    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:29.366465 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:29.366418    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:32.075071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:32.075021    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:32.265212 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:32.265171    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:32.446599 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:32.446498    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:34.098083 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:34.098030    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:35.276196 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:35.276158    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:35.449995 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:35.449959    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:39.361302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:39.361219    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:43.080394 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:43.080346    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:44.275592 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:44.275544    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:44.851639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:44.851605    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:45.042526 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:45.042488    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:45.296920 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:45.296879    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:48.888830 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:48.888791    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:49.135039 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:49.135002    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:49.319935 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:49.319898    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:52.119786 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:52.119734    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:52.300373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:52.300330    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:57.359815 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:26:57.359749    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:26:57.594930 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:57.594896    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:58.383575 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:58.383534    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:26:59.705970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:26:59.705935    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:06.862828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:06.862762    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:12.187149 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:12.187104    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:17.247603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:17.247565    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:19.564092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:19.564018    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:21.019721 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:21.019684    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:27:21.242672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:21.242629    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:21.461038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:21.460933    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:22.324257 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:22.324214    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:22.507675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:22.507631    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:23.620442 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:23.620384    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:25.316660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:25.316621    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:27.360960 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:27:27.360922    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:27:27.490950 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:27.490830    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:29.437857 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:29.437821    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj_7035711a-f7cb-48f5-a7fc-d66673ea55c2/payload-processing/0.log"
Nov 02 20:27:31.005366 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:31.005319    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:31.549673 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:31.549632    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:32.080124 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.080077    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"]
Nov 02 20:27:32.080819 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.080789    5224 cpu_manager.go:414] "containerMap: removing container" podUID="b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08" containerName="maas-api"
Nov 02 20:27:32.080819 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.080817    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08" containerName="maas-api"
Nov 02 20:27:32.081005 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.080850    5224 cpu_manager.go:414] "containerMap: removing container" podUID="2e99203c-3219-44e7-9af6-d3de71bbc242" containerName="payload-pre-processing"
Nov 02 20:27:32.081005 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.080856    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e99203c-3219-44e7-9af6-d3de71bbc242" containerName="payload-pre-processing"
Nov 02 20:27:32.081005 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.080871    5224 cpu_manager.go:414] "containerMap: removing container" podUID="ed0d9cbd-fa51-44ea-8333-2b7db2337e89" containerName="istio-proxy"
Nov 02 20:27:32.081005 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.080877    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0d9cbd-fa51-44ea-8333-2b7db2337e89" containerName="istio-proxy"
Nov 02 20:27:32.081005 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.080889    5224 cpu_manager.go:414] "containerMap: removing container" podUID="8deecf9a-e43f-42bb-bc42-6924b0272459" containerName="payload-processing"
Nov 02 20:27:32.081005 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.080895    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8deecf9a-e43f-42bb-bc42-6924b0272459" containerName="payload-processing"
Nov 02 20:27:32.081005 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.080974    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e99203c-3219-44e7-9af6-d3de71bbc242" containerName="payload-pre-processing"
Nov 02 20:27:32.081005 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.080985    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0d9cbd-fa51-44ea-8333-2b7db2337e89" containerName="istio-proxy"
Nov 02 20:27:32.081005 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.080993    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="8deecf9a-e43f-42bb-bc42-6924b0272459" containerName="payload-processing"
Nov 02 20:27:32.081005 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.081000    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b36f5cc9-98ed-4eaa-8cd5-cc3b8dd73e08" containerName="maas-api"
Nov 02 20:27:32.084826 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.084760    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.140644 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.140607    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"]
Nov 02 20:27:32.190620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.190585    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-7cad76-serving-cert\""
Nov 02 20:27:32.203801 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.203740    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-metrics-certs\") pod \"maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.203977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.203899    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rwv\" (UniqueName: \"kubernetes.io/projected/0543f1ee-7bcb-491e-94b0-996ec5133c2f-kube-api-access-56rwv\") pod \"maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.203977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.203961    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0543f1ee-7bcb-491e-94b0-996ec5133c2f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.204054 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.204031    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.305383 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.305336    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rwv\" (UniqueName: \"kubernetes.io/projected/0543f1ee-7bcb-491e-94b0-996ec5133c2f-kube-api-access-56rwv\") pod \"maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.305611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.305417    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0543f1ee-7bcb-491e-94b0-996ec5133c2f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.305611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.305465    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.305611 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:27:32.305598    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-serving-cert: secret "maas-api-e2e-praxis-a-7cad76-serving-cert" not found
Nov 02 20:27:32.305829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.305610    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-metrics-certs\") pod \"maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.305829 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:27:32.305688    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-maas-api-tls podName:0543f1ee-7bcb-491e-94b0-996ec5133c2f nodeName:}" failed. No retries permitted until 2026-11-02 20:27:32.805664738 +0000 UTC m=+2333.803353331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-maas-api-tls") pod "maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb" (UID: "0543f1ee-7bcb-491e-94b0-996ec5133c2f") : secret "maas-api-e2e-praxis-a-7cad76-serving-cert" not found
Nov 02 20:27:32.306364 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.306345    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0543f1ee-7bcb-491e-94b0-996ec5133c2f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.308100 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.308080    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-metrics-certs\") pod \"maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.345177 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.345082    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:32.368278 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.368237    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rwv\" (UniqueName: \"kubernetes.io/projected/0543f1ee-7bcb-491e-94b0-996ec5133c2f-kube-api-access-56rwv\") pod \"maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.572822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.572753    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:32.771560 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.771509    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:32.811180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.811131    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.813736 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.813709    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:32.998628 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:32.998589    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:33.381439 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:27:33.381401    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0543f1ee_7bcb_491e_94b0_996ec5133c2f.slice/crio-c28baf91a065b29cf5dbd6a2f6060e0966c4aa367a1d421000c2c4c2b70bcc53 WatchSource:0}: Error finding container c28baf91a065b29cf5dbd6a2f6060e0966c4aa367a1d421000c2c4c2b70bcc53: Status 404 returned error can't find the container with id c28baf91a065b29cf5dbd6a2f6060e0966c4aa367a1d421000c2c4c2b70bcc53
Nov 02 20:27:33.383277 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:33.383255    5224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:27:33.383650 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:33.383630    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"]
Nov 02 20:27:34.198297 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:34.198254    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb" event={"ID":"0543f1ee-7bcb-491e-94b0-996ec5133c2f","Type":"ContainerStarted","Data":"c28baf91a065b29cf5dbd6a2f6060e0966c4aa367a1d421000c2c4c2b70bcc53"}
Nov 02 20:27:35.205490 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:35.205437    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb" event={"ID":"0543f1ee-7bcb-491e-94b0-996ec5133c2f","Type":"ContainerStarted","Data":"ea61d1281fde7ede3307f357275aceecc0e7c2717aaf35e04d43782cac34164a"}
Nov 02 20:27:35.205955 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:35.205513    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:35.324035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:35.323977    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb" podStartSLOduration=2.392619757 podStartE2EDuration="3.323961244s" totalImagesPullingTime="931.341487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:35.323652456 +0000 UTC m=+2336.321341065" watchObservedRunningTime="2026-11-02 20:27:35.323961244 +0000 UTC m=+2336.321649853"
Nov 02 20:27:36.470391 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:36.470351    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"]
Nov 02 20:27:36.591107 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:36.591069    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:38.001046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:38.001007    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:41.221576 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:41.221545    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:27:43.044735 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:43.044688    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:47.859141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:47.859102    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:48.035566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:48.035516    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:51.491942 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.491900    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8" podUID="65f01ab5-ac30-406c-b8a6-63a76e042836" containerName="main" containerID="cri-o://ea259808605e1261f071baee6771c5eb987019f138c623850208711f24c97159" gracePeriod=45
Nov 02 20:27:51.763085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.763053    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:27:51.849097 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.849060    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f01ab5-ac30-406c-b8a6-63a76e042836-kube-api-access-9rhwh\" (UniqueName: \"kubernetes.io/projected/65f01ab5-ac30-406c-b8a6-63a76e042836-kube-api-access-9rhwh\") pod \"65f01ab5-ac30-406c-b8a6-63a76e042836\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") "
Nov 02 20:27:51.849324 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.849120    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-dshm\") pod \"65f01ab5-ac30-406c-b8a6-63a76e042836\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") "
Nov 02 20:27:51.849324 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.849156    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-tmp-dir\") pod \"65f01ab5-ac30-406c-b8a6-63a76e042836\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") "
Nov 02 20:27:51.849324 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.849219    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-model-cache\") pod \"65f01ab5-ac30-406c-b8a6-63a76e042836\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") "
Nov 02 20:27:51.849324 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.849291    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-home\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-home\") pod \"65f01ab5-ac30-406c-b8a6-63a76e042836\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") "
Nov 02 20:27:51.849559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.849338    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65f01ab5-ac30-406c-b8a6-63a76e042836-tls-certs\" (UniqueName: \"kubernetes.io/secret/65f01ab5-ac30-406c-b8a6-63a76e042836-tls-certs\") pod \"65f01ab5-ac30-406c-b8a6-63a76e042836\" (UID: \"65f01ab5-ac30-406c-b8a6-63a76e042836\") "
Nov 02 20:27:51.849559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.849484    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-tmp-dir" pod "65f01ab5-ac30-406c-b8a6-63a76e042836" (UID: "65f01ab5-ac30-406c-b8a6-63a76e042836"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:27:51.849559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.849524    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-model-cache" pod "65f01ab5-ac30-406c-b8a6-63a76e042836" (UID: "65f01ab5-ac30-406c-b8a6-63a76e042836"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:27:51.849678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.849564    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-home" pod "65f01ab5-ac30-406c-b8a6-63a76e042836" (UID: "65f01ab5-ac30-406c-b8a6-63a76e042836"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:27:51.849948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.849913    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:27:51.849948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.849936    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:27:51.849948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.849950    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:27:51.851627 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.851602    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f01ab5-ac30-406c-b8a6-63a76e042836-tls-certs" pod "65f01ab5-ac30-406c-b8a6-63a76e042836" (UID: "65f01ab5-ac30-406c-b8a6-63a76e042836"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:27:51.851627 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.851619    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f01ab5-ac30-406c-b8a6-63a76e042836-kube-api-access-9rhwh" pod "65f01ab5-ac30-406c-b8a6-63a76e042836" (UID: "65f01ab5-ac30-406c-b8a6-63a76e042836"). InnerVolumeSpecName "kube-api-access-9rhwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:27:51.851827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.851608    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-dshm" pod "65f01ab5-ac30-406c-b8a6-63a76e042836" (UID: "65f01ab5-ac30-406c-b8a6-63a76e042836"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:27:51.950864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.950824    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rhwh\" (UniqueName: \"kubernetes.io/projected/65f01ab5-ac30-406c-b8a6-63a76e042836-kube-api-access-9rhwh\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:27:51.950864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.950861    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f01ab5-ac30-406c-b8a6-63a76e042836-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:27:51.950864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:51.950873    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f01ab5-ac30-406c-b8a6-63a76e042836-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:27:52.312458 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:52.312425    5224 generic.go:356] "Generic (PLEG): container finished" podID="65f01ab5-ac30-406c-b8a6-63a76e042836" containerID="ea259808605e1261f071baee6771c5eb987019f138c623850208711f24c97159" exitCode=0
Nov 02 20:27:52.312673 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:52.312473    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"
Nov 02 20:27:52.312673 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:52.312524    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8" event={"ID":"65f01ab5-ac30-406c-b8a6-63a76e042836","Type":"ContainerDied","Data":"ea259808605e1261f071baee6771c5eb987019f138c623850208711f24c97159"}
Nov 02 20:27:52.312673 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:52.312570    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8" event={"ID":"65f01ab5-ac30-406c-b8a6-63a76e042836","Type":"ContainerDied","Data":"90d4f6b7cab3a0325aaff0d1af7fa1d56144e5d7ec8c48421559158aa58ea6ae"}
Nov 02 20:27:52.312673 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:52.312591    5224 scope.go:122] "RemoveContainer" containerID="ea259808605e1261f071baee6771c5eb987019f138c623850208711f24c97159"
Nov 02 20:27:52.322560 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:52.322539    5224 scope.go:122] "RemoveContainer" containerID="ea259808605e1261f071baee6771c5eb987019f138c623850208711f24c97159"
Nov 02 20:27:52.322845 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:27:52.322827    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea259808605e1261f071baee6771c5eb987019f138c623850208711f24c97159\": container with ID starting with ea259808605e1261f071baee6771c5eb987019f138c623850208711f24c97159 not found: ID does not exist" containerID="ea259808605e1261f071baee6771c5eb987019f138c623850208711f24c97159"
Nov 02 20:27:52.322902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:52.322854    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea259808605e1261f071baee6771c5eb987019f138c623850208711f24c97159"} err="failed to get container status \"ea259808605e1261f071baee6771c5eb987019f138c623850208711f24c97159\": rpc error: code = NotFound desc = could not find container \"ea259808605e1261f071baee6771c5eb987019f138c623850208711f24c97159\": container with ID starting with ea259808605e1261f071baee6771c5eb987019f138c623850208711f24c97159 not found: ID does not exist"
Nov 02 20:27:52.479667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:52.479626    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"]
Nov 02 20:27:52.483761 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:52.483734    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-e87e9a/ipp-route-e87e9a-kserve-76788fb5bf-lnnr8"]
Nov 02 20:27:52.856520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:52.856481    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:53.167956 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:53.167875    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f01ab5-ac30-406c-b8a6-63a76e042836" path="/var/lib/kubelet/pods/65f01ab5-ac30-406c-b8a6-63a76e042836/volumes"
Nov 02 20:27:56.574671 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:56.574627    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:27:57.360903 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:27:57.360864    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:27:58.952022 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:27:58.951984    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:02.871109 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:02.871068    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:09.291103 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:09.291064    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:09.469557 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:09.469521    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:09.747842 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:09.747728    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:13.405447 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:13.405396    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:13.743858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:13.743721    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:13.929410 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:13.929375    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:14.464947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:14.464910    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:14.639371 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:14.639327    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:16.095969 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:16.095917    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:18.050078 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:18.050032    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:18.352526 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:18.352423    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:18.646023 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:18.645934    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:28:19.295559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:19.295500    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:19.668512 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:19.668420    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:20.286965 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:20.286926    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:21.081379 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:21.081338    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:22.020064 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:22.020022    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:23.510318 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:23.510205    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:24.938385 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:24.938338    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:26.784369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:26.784323    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:27.359682 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:28:27.359632    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:28:27.966483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:27.966441    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:29.972083 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:29.972036    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:32.304170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:32.304128    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:32.497414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:32.497368    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:33.833584 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:33.833540    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:34.323014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:34.322972    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:35.193874 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:35.193837    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:35.812379 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:35.812334    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:36.427855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:36.427717    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:36.838362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:36.838322    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:40.362394 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:40.362344    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:41.054761 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:41.054721    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:41.244900 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:41.244826    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:41.471027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:41.470929    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:44.212119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:44.212079    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:44.400338 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:44.400296    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:45.805445 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:45.805388    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:50.784483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:50.784432    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:54.117525 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:54.117483    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:55.242652 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:55.242608    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:55.801541 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:55.801321    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:56.103467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:56.103373    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:56.326063 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:56.326017    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:57.359700 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:28:57.359659    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:28:58.267299 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:58.267261    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:28:59.052158 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:28:59.052114    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:00.177204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:00.177168    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:00.355153 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:00.355115    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:01.575739 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:01.575684    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:03.279581 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:03.279543    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:03.499185 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:03.499129    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:03.672862 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:03.672752    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:06.753600 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:06.753562    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:10.250040 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:10.249993    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:29:10.330816 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:10.330747    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:12.093557 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:12.093506    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:13.265842 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:13.265802    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:16.754599 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:16.754553    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:17.095290 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.095251    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"]
Nov 02 20:29:17.095506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.095484    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj" podUID="7035711a-f7cb-48f5-a7fc-d66673ea55c2" containerName="payload-processing" containerID="cri-o://399bcee2f496afe7f643f5614da41832972ac594c8f0796dc63fec67d7000896" gracePeriod=30
Nov 02 20:29:17.203976 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.203942    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"]
Nov 02 20:29:17.204256 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.204203    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj" podUID="44fc8090-e113-4fdc-a497-20d55f78b354" containerName="payload-pre-processing" containerID="cri-o://4f954375cc4f8711b84351b41b2eeb394078d04491d1a11c1afc5fed5a1e9375" gracePeriod=30
Nov 02 20:29:17.324507 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.324466    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:17.413177 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.413144    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"
Nov 02 20:29:17.489015 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.488987    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"
Nov 02 20:29:17.499886 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.499850    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:17.554488 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.554446    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7035711a-f7cb-48f5-a7fc-d66673ea55c2-kube-api-access-dhj8j\" (UniqueName: \"kubernetes.io/projected/7035711a-f7cb-48f5-a7fc-d66673ea55c2-kube-api-access-dhj8j\") pod \"7035711a-f7cb-48f5-a7fc-d66673ea55c2\" (UID: \"7035711a-f7cb-48f5-a7fc-d66673ea55c2\") "
Nov 02 20:29:17.554708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.554616    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7035711a-f7cb-48f5-a7fc-d66673ea55c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7035711a-f7cb-48f5-a7fc-d66673ea55c2-plugins-config-volume\") pod \"7035711a-f7cb-48f5-a7fc-d66673ea55c2\" (UID: \"7035711a-f7cb-48f5-a7fc-d66673ea55c2\") "
Nov 02 20:29:17.555047 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.555018    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7035711a-f7cb-48f5-a7fc-d66673ea55c2-plugins-config-volume" pod "7035711a-f7cb-48f5-a7fc-d66673ea55c2" (UID: "7035711a-f7cb-48f5-a7fc-d66673ea55c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:17.556852 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.556820    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7035711a-f7cb-48f5-a7fc-d66673ea55c2-kube-api-access-dhj8j" pod "7035711a-f7cb-48f5-a7fc-d66673ea55c2" (UID: "7035711a-f7cb-48f5-a7fc-d66673ea55c2"). InnerVolumeSpecName "kube-api-access-dhj8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:17.670838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.657147    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fc8090-e113-4fdc-a497-20d55f78b354-kube-api-access-vgqrz\" (UniqueName: \"kubernetes.io/projected/44fc8090-e113-4fdc-a497-20d55f78b354-kube-api-access-vgqrz\") pod \"44fc8090-e113-4fdc-a497-20d55f78b354\" (UID: \"44fc8090-e113-4fdc-a497-20d55f78b354\") "
Nov 02 20:29:17.670838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.657395    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44fc8090-e113-4fdc-a497-20d55f78b354-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fc8090-e113-4fdc-a497-20d55f78b354-plugins-config-volume\") pod \"44fc8090-e113-4fdc-a497-20d55f78b354\" (UID: \"44fc8090-e113-4fdc-a497-20d55f78b354\") "
Nov 02 20:29:17.670838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.657856    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhj8j\" (UniqueName: \"kubernetes.io/projected/7035711a-f7cb-48f5-a7fc-d66673ea55c2-kube-api-access-dhj8j\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:17.670838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.657880    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7035711a-f7cb-48f5-a7fc-d66673ea55c2-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:17.670838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.664013    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44fc8090-e113-4fdc-a497-20d55f78b354-plugins-config-volume" pod "44fc8090-e113-4fdc-a497-20d55f78b354" (UID: "44fc8090-e113-4fdc-a497-20d55f78b354"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:17.670838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.667139    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fc8090-e113-4fdc-a497-20d55f78b354-kube-api-access-vgqrz" pod "44fc8090-e113-4fdc-a497-20d55f78b354" (UID: "44fc8090-e113-4fdc-a497-20d55f78b354"). InnerVolumeSpecName "kube-api-access-vgqrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:17.759332 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.759290    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fc8090-e113-4fdc-a497-20d55f78b354-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:17.759332 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.759325    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgqrz\" (UniqueName: \"kubernetes.io/projected/44fc8090-e113-4fdc-a497-20d55f78b354-kube-api-access-vgqrz\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:17.849377 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.849333    5224 generic.go:356] "Generic (PLEG): container finished" podID="7035711a-f7cb-48f5-a7fc-d66673ea55c2" containerID="399bcee2f496afe7f643f5614da41832972ac594c8f0796dc63fec67d7000896" exitCode=0
Nov 02 20:29:17.849377 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.849381    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"
Nov 02 20:29:17.849635 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.849423    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj" event={"ID":"7035711a-f7cb-48f5-a7fc-d66673ea55c2","Type":"ContainerDied","Data":"399bcee2f496afe7f643f5614da41832972ac594c8f0796dc63fec67d7000896"}
Nov 02 20:29:17.849635 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.849475    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj" event={"ID":"7035711a-f7cb-48f5-a7fc-d66673ea55c2","Type":"ContainerDied","Data":"5f9f04156077cde8ce3140f3c901ffeb377e56933a19abeba13cd7c546ae76d2"}
Nov 02 20:29:17.849635 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.849498    5224 scope.go:122] "RemoveContainer" containerID="399bcee2f496afe7f643f5614da41832972ac594c8f0796dc63fec67d7000896"
Nov 02 20:29:17.850916 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.850888    5224 generic.go:356] "Generic (PLEG): container finished" podID="44fc8090-e113-4fdc-a497-20d55f78b354" containerID="4f954375cc4f8711b84351b41b2eeb394078d04491d1a11c1afc5fed5a1e9375" exitCode=0
Nov 02 20:29:17.851044 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.850938    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"
Nov 02 20:29:17.851044 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.850934    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj" event={"ID":"44fc8090-e113-4fdc-a497-20d55f78b354","Type":"ContainerDied","Data":"4f954375cc4f8711b84351b41b2eeb394078d04491d1a11c1afc5fed5a1e9375"}
Nov 02 20:29:17.851169 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.851065    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj" event={"ID":"44fc8090-e113-4fdc-a497-20d55f78b354","Type":"ContainerDied","Data":"832aea3729bd1d5264f77843e1512a44813ed28808918e8f61b4622eea09cfd2"}
Nov 02 20:29:17.862514 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.862157    5224 scope.go:122] "RemoveContainer" containerID="399bcee2f496afe7f643f5614da41832972ac594c8f0796dc63fec67d7000896"
Nov 02 20:29:17.862602 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:29:17.862531    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399bcee2f496afe7f643f5614da41832972ac594c8f0796dc63fec67d7000896\": container with ID starting with 399bcee2f496afe7f643f5614da41832972ac594c8f0796dc63fec67d7000896 not found: ID does not exist" containerID="399bcee2f496afe7f643f5614da41832972ac594c8f0796dc63fec67d7000896"
Nov 02 20:29:17.862602 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.862560    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399bcee2f496afe7f643f5614da41832972ac594c8f0796dc63fec67d7000896"} err="failed to get container status \"399bcee2f496afe7f643f5614da41832972ac594c8f0796dc63fec67d7000896\": rpc error: code = NotFound desc = could not find container \"399bcee2f496afe7f643f5614da41832972ac594c8f0796dc63fec67d7000896\": container with ID starting with 399bcee2f496afe7f643f5614da41832972ac594c8f0796dc63fec67d7000896 not found: ID does not exist"
Nov 02 20:29:17.862602 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.862578    5224 scope.go:122] "RemoveContainer" containerID="4f954375cc4f8711b84351b41b2eeb394078d04491d1a11c1afc5fed5a1e9375"
Nov 02 20:29:17.872693 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.872670    5224 scope.go:122] "RemoveContainer" containerID="4f954375cc4f8711b84351b41b2eeb394078d04491d1a11c1afc5fed5a1e9375"
Nov 02 20:29:17.872961 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:29:17.872941    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f954375cc4f8711b84351b41b2eeb394078d04491d1a11c1afc5fed5a1e9375\": container with ID starting with 4f954375cc4f8711b84351b41b2eeb394078d04491d1a11c1afc5fed5a1e9375 not found: ID does not exist" containerID="4f954375cc4f8711b84351b41b2eeb394078d04491d1a11c1afc5fed5a1e9375"
Nov 02 20:29:17.873023 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:17.872972    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f954375cc4f8711b84351b41b2eeb394078d04491d1a11c1afc5fed5a1e9375"} err="failed to get container status \"4f954375cc4f8711b84351b41b2eeb394078d04491d1a11c1afc5fed5a1e9375\": rpc error: code = NotFound desc = could not find container \"4f954375cc4f8711b84351b41b2eeb394078d04491d1a11c1afc5fed5a1e9375\": container with ID starting with 4f954375cc4f8711b84351b41b2eeb394078d04491d1a11c1afc5fed5a1e9375 not found: ID does not exist"
Nov 02 20:29:18.016801 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:18.016749    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"]
Nov 02 20:29:18.019063 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:18.019043    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e87e9a-57c74f6cd8-lszzj"]
Nov 02 20:29:18.129321 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:18.129283    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"]
Nov 02 20:29:18.133726 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:18.133698    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e87e9a-8856f78c-sc9dj"]
Nov 02 20:29:18.604146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:18.604091    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:18.781524 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:18.781487    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:19.050092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:19.050059    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"]
Nov 02 20:29:19.050323 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:19.050291    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy" containerID="cri-o://3daa8e804dfccdff746f2c698cf5fd1a94391738d6855daf531650f895a6c20d" gracePeriod=30
Nov 02 20:29:19.089589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:19.089549    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:19.195791 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:19.192599    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44fc8090-e113-4fdc-a497-20d55f78b354" path="/var/lib/kubelet/pods/44fc8090-e113-4fdc-a497-20d55f78b354/volumes"
Nov 02 20:29:19.210794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:19.203736    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7035711a-f7cb-48f5-a7fc-d66673ea55c2" path="/var/lib/kubelet/pods/7035711a-f7cb-48f5-a7fc-d66673ea55c2/volumes"
Nov 02 20:29:19.263176 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:19.263139    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:19.877189 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:19.877147    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:20.098885 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:20.098844    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:22.274632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:22.274591    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:22.454276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:22.454235    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:23.544263 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:23.544220    5224 patch_prober.go:29] interesting pod/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:29:23.544708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:23.544293    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:29:24.197852 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.197806    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:24.322306 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.322220    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:29:24.367473 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.367436    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-podinfo\") pod \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") "
Nov 02 20:29:24.367666 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.367497    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-envoy\") pod \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") "
Nov 02 20:29:24.367666 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.367526    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-kube-api-access-z9cf7\" (UniqueName: \"kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-kube-api-access-z9cf7\") pod \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") "
Nov 02 20:29:24.367666 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.367597    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istiod-ca-cert\") pod \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") "
Nov 02 20:29:24.367666 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.367634    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-socket\") pod \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") "
Nov 02 20:29:24.367915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.367683    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-token\" (UniqueName: \"kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-token\") pod \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") "
Nov 02 20:29:24.367915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.367701    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-credential-socket\") pod \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") "
Nov 02 20:29:24.367915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.367726    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-certs\") pod \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") "
Nov 02 20:29:24.367915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.367751    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-data\") pod \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\" (UID: \"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e\") "
Nov 02 20:29:24.368125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.368098    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-socket" pod "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" (UID: "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:24.368191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.368028    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istiod-ca-cert" pod "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" (UID: "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:24.368293 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.368271    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:24.368477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.368456    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:24.368477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.368291    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-certs" pod "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" (UID: "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:24.368605 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.368312    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-data" pod "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" (UID: "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:24.368605 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.368481    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-credential-socket" pod "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" (UID: "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:24.370311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.370286    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-podinfo" pod "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" (UID: "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:24.370311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.370300    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-envoy" pod "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" (UID: "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:24.370470 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.370341    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-kube-api-access-z9cf7" pod "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" (UID: "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e"). InnerVolumeSpecName "kube-api-access-z9cf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:24.370470 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.370365    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-token" pod "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" (UID: "8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:24.371686 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.371649    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:24.470065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.470025    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:24.470065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.470059    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:24.470065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.470071    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9cf7\" (UniqueName: \"kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-kube-api-access-z9cf7\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:24.470319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.470083    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:24.470319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.470092    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:24.470319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.470101    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:24.470319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.470127    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:24.617548 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.617444    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:24.898892 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.898799    5224 generic.go:356] "Generic (PLEG): container finished" podID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerID="3daa8e804dfccdff746f2c698cf5fd1a94391738d6855daf531650f895a6c20d" exitCode=0
Nov 02 20:29:24.898892 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.898853    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"
Nov 02 20:29:24.898892 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.898864    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" event={"ID":"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e","Type":"ContainerDied","Data":"3daa8e804dfccdff746f2c698cf5fd1a94391738d6855daf531650f895a6c20d"}
Nov 02 20:29:24.899180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.898959    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f" event={"ID":"8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e","Type":"ContainerDied","Data":"9c830e53ef36f6f315056c9398dfd980b742a17d5a184be7afa87365e547489f"}
Nov 02 20:29:24.899180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.898995    5224 scope.go:122] "RemoveContainer" containerID="3daa8e804dfccdff746f2c698cf5fd1a94391738d6855daf531650f895a6c20d"
Nov 02 20:29:24.910619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.910589    5224 scope.go:122] "RemoveContainer" containerID="beae7518d34ac2db33d20bb221a7d303fee816ede3b0432c0a01142b1ca8abbf"
Nov 02 20:29:24.921871 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.921846    5224 scope.go:122] "RemoveContainer" containerID="3daa8e804dfccdff746f2c698cf5fd1a94391738d6855daf531650f895a6c20d"
Nov 02 20:29:24.922208 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:29:24.922189    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3daa8e804dfccdff746f2c698cf5fd1a94391738d6855daf531650f895a6c20d\": container with ID starting with 3daa8e804dfccdff746f2c698cf5fd1a94391738d6855daf531650f895a6c20d not found: ID does not exist" containerID="3daa8e804dfccdff746f2c698cf5fd1a94391738d6855daf531650f895a6c20d"
Nov 02 20:29:24.922322 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.922218    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3daa8e804dfccdff746f2c698cf5fd1a94391738d6855daf531650f895a6c20d"} err="failed to get container status \"3daa8e804dfccdff746f2c698cf5fd1a94391738d6855daf531650f895a6c20d\": rpc error: code = NotFound desc = could not find container \"3daa8e804dfccdff746f2c698cf5fd1a94391738d6855daf531650f895a6c20d\": container with ID starting with 3daa8e804dfccdff746f2c698cf5fd1a94391738d6855daf531650f895a6c20d not found: ID does not exist"
Nov 02 20:29:24.922322 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.922237    5224 scope.go:122] "RemoveContainer" containerID="beae7518d34ac2db33d20bb221a7d303fee816ede3b0432c0a01142b1ca8abbf"
Nov 02 20:29:24.922475 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:29:24.922459    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beae7518d34ac2db33d20bb221a7d303fee816ede3b0432c0a01142b1ca8abbf\": container with ID starting with beae7518d34ac2db33d20bb221a7d303fee816ede3b0432c0a01142b1ca8abbf not found: ID does not exist" containerID="beae7518d34ac2db33d20bb221a7d303fee816ede3b0432c0a01142b1ca8abbf"
Nov 02 20:29:24.922534 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:24.922478    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beae7518d34ac2db33d20bb221a7d303fee816ede3b0432c0a01142b1ca8abbf"} err="failed to get container status \"beae7518d34ac2db33d20bb221a7d303fee816ede3b0432c0a01142b1ca8abbf\": rpc error: code = NotFound desc = could not find container \"beae7518d34ac2db33d20bb221a7d303fee816ede3b0432c0a01142b1ca8abbf\": container with ID starting with beae7518d34ac2db33d20bb221a7d303fee816ede3b0432c0a01142b1ca8abbf not found: ID does not exist"
Nov 02 20:29:25.078189 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:25.078149    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"]
Nov 02 20:29:25.130727 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:25.130685    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e87e9a-openshift-default-7dd757784f-7nk2f"]
Nov 02 20:29:25.167659 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:25.167580    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" path="/var/lib/kubelet/pods/8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e/volumes"
Nov 02 20:29:27.360506 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:29:27.360469    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:29:27.413172 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:27.413135    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:29.292470 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:29.292417    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:29.654707 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:29.654612    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"]
Nov 02 20:29:29.654918 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:29.654844    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd" podUID="98a0e46b-fc81-4841-a517-b56e0a54061d" containerName="istio-proxy" containerID="cri-o://9496cc314281076e4994364dd1e88a90e3e64b73ec6782c7782e5b88453ac809" gracePeriod=30
Nov 02 20:29:29.766254 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:29.766215    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"]
Nov 02 20:29:29.766480 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:29.766455    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" podUID="9f35ea3d-b5b9-47b2-800d-78cc466867af" containerName="istio-proxy" containerID="cri-o://5cf2aa375b80445714c151924b8c4b658ba40ff4006dd8036cd44f3bc4e1b167" gracePeriod=30
Nov 02 20:29:30.046444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:30.046394    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:31.549336 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:31.549292    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:32.338554 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:32.338509    5224 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:29:32.338749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:32.338576    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" podUID="9f35ea3d-b5b9-47b2-800d-78cc466867af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:29:34.945843 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:34.945804    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:29:34.970581 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:34.970550    5224 generic.go:356] "Generic (PLEG): container finished" podID="9f35ea3d-b5b9-47b2-800d-78cc466867af" containerID="5cf2aa375b80445714c151924b8c4b658ba40ff4006dd8036cd44f3bc4e1b167" exitCode=0
Nov 02 20:29:34.970786 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:34.970608    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" event={"ID":"9f35ea3d-b5b9-47b2-800d-78cc466867af","Type":"ContainerDied","Data":"5cf2aa375b80445714c151924b8c4b658ba40ff4006dd8036cd44f3bc4e1b167"}
Nov 02 20:29:34.972218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:34.972184    5224 generic.go:356] "Generic (PLEG): container finished" podID="98a0e46b-fc81-4841-a517-b56e0a54061d" containerID="9496cc314281076e4994364dd1e88a90e3e64b73ec6782c7782e5b88453ac809" exitCode=0
Nov 02 20:29:34.972364 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:34.972252    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"
Nov 02 20:29:34.972364 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:34.972280    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd" event={"ID":"98a0e46b-fc81-4841-a517-b56e0a54061d","Type":"ContainerDied","Data":"9496cc314281076e4994364dd1e88a90e3e64b73ec6782c7782e5b88453ac809"}
Nov 02 20:29:34.972364 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:34.972337    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd" event={"ID":"98a0e46b-fc81-4841-a517-b56e0a54061d","Type":"ContainerDied","Data":"2e74e799878a90cb1b1878cd5f72a72a873a969b91d5d222b4ef5ec6f3389872"}
Nov 02 20:29:34.972542 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:34.972367    5224 scope.go:122] "RemoveContainer" containerID="9496cc314281076e4994364dd1e88a90e3e64b73ec6782c7782e5b88453ac809"
Nov 02 20:29:34.984106 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:34.983916    5224 scope.go:122] "RemoveContainer" containerID="9496cc314281076e4994364dd1e88a90e3e64b73ec6782c7782e5b88453ac809"
Nov 02 20:29:34.984592 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:29:34.984559    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9496cc314281076e4994364dd1e88a90e3e64b73ec6782c7782e5b88453ac809\": container with ID starting with 9496cc314281076e4994364dd1e88a90e3e64b73ec6782c7782e5b88453ac809 not found: ID does not exist" containerID="9496cc314281076e4994364dd1e88a90e3e64b73ec6782c7782e5b88453ac809"
Nov 02 20:29:34.984720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:34.984605    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9496cc314281076e4994364dd1e88a90e3e64b73ec6782c7782e5b88453ac809"} err="failed to get container status \"9496cc314281076e4994364dd1e88a90e3e64b73ec6782c7782e5b88453ac809\": rpc error: code = NotFound desc = could not find container \"9496cc314281076e4994364dd1e88a90e3e64b73ec6782c7782e5b88453ac809\": container with ID starting with 9496cc314281076e4994364dd1e88a90e3e64b73ec6782c7782e5b88453ac809 not found: ID does not exist"
Nov 02 20:29:35.052307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.052272    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:35.059593 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.059568    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:29:35.096363 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.096330    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-kube-api-access-drbhb\" (UniqueName: \"kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-kube-api-access-drbhb\") pod \"98a0e46b-fc81-4841-a517-b56e0a54061d\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") "
Nov 02 20:29:35.096560 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.096427    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-certs\") pod \"98a0e46b-fc81-4841-a517-b56e0a54061d\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") "
Nov 02 20:29:35.096560 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.096456    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-socket\") pod \"98a0e46b-fc81-4841-a517-b56e0a54061d\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") "
Nov 02 20:29:35.096560 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.096481    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-token\" (UniqueName: \"kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-token\") pod \"98a0e46b-fc81-4841-a517-b56e0a54061d\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") "
Nov 02 20:29:35.096560 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.096518    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-data\") pod \"98a0e46b-fc81-4841-a517-b56e0a54061d\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") "
Nov 02 20:29:35.096860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.096595    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-credential-socket\") pod \"98a0e46b-fc81-4841-a517-b56e0a54061d\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") "
Nov 02 20:29:35.096860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.096643    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a0e46b-fc81-4841-a517-b56e0a54061d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a0e46b-fc81-4841-a517-b56e0a54061d-istiod-ca-cert\") pod \"98a0e46b-fc81-4841-a517-b56e0a54061d\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") "
Nov 02 20:29:35.096860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.096676    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-envoy\") pod \"98a0e46b-fc81-4841-a517-b56e0a54061d\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") "
Nov 02 20:29:35.096860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.096717    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-podinfo\") pod \"98a0e46b-fc81-4841-a517-b56e0a54061d\" (UID: \"98a0e46b-fc81-4841-a517-b56e0a54061d\") "
Nov 02 20:29:35.096860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.096740    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-certs" pod "98a0e46b-fc81-4841-a517-b56e0a54061d" (UID: "98a0e46b-fc81-4841-a517-b56e0a54061d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:35.097150 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.096855    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-socket" pod "98a0e46b-fc81-4841-a517-b56e0a54061d" (UID: "98a0e46b-fc81-4841-a517-b56e0a54061d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:35.097150 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.097015    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-credential-socket" pod "98a0e46b-fc81-4841-a517-b56e0a54061d" (UID: "98a0e46b-fc81-4841-a517-b56e0a54061d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:35.097150 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.097131    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-data" pod "98a0e46b-fc81-4841-a517-b56e0a54061d" (UID: "98a0e46b-fc81-4841-a517-b56e0a54061d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:35.097279 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.097173    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a0e46b-fc81-4841-a517-b56e0a54061d-istiod-ca-cert" pod "98a0e46b-fc81-4841-a517-b56e0a54061d" (UID: "98a0e46b-fc81-4841-a517-b56e0a54061d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:35.097344 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.097300    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.097344 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.097315    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a0e46b-fc81-4841-a517-b56e0a54061d-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.097344 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.097325    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.097344 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.097333    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.097344 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.097343    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.098822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.098793    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-token" pod "98a0e46b-fc81-4841-a517-b56e0a54061d" (UID: "98a0e46b-fc81-4841-a517-b56e0a54061d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:35.098941 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.098904    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-podinfo" pod "98a0e46b-fc81-4841-a517-b56e0a54061d" (UID: "98a0e46b-fc81-4841-a517-b56e0a54061d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:35.099125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.099111    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-kube-api-access-drbhb" pod "98a0e46b-fc81-4841-a517-b56e0a54061d" (UID: "98a0e46b-fc81-4841-a517-b56e0a54061d"). InnerVolumeSpecName "kube-api-access-drbhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:35.099206 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.099184    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-envoy" pod "98a0e46b-fc81-4841-a517-b56e0a54061d" (UID: "98a0e46b-fc81-4841-a517-b56e0a54061d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:35.198237 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198189    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f35ea3d-b5b9-47b2-800d-78cc466867af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f35ea3d-b5b9-47b2-800d-78cc466867af-istiod-ca-cert\") pod \"9f35ea3d-b5b9-47b2-800d-78cc466867af\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") "
Nov 02 20:29:35.198458 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198264    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-envoy\") pod \"9f35ea3d-b5b9-47b2-800d-78cc466867af\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") "
Nov 02 20:29:35.198458 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198353    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-podinfo\") pod \"9f35ea3d-b5b9-47b2-800d-78cc466867af\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") "
Nov 02 20:29:35.198458 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198404    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-data\") pod \"9f35ea3d-b5b9-47b2-800d-78cc466867af\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") "
Nov 02 20:29:35.198635 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198481    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-socket\") pod \"9f35ea3d-b5b9-47b2-800d-78cc466867af\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") "
Nov 02 20:29:35.198635 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198531    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-certs\") pod \"9f35ea3d-b5b9-47b2-800d-78cc466867af\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") "
Nov 02 20:29:35.198635 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198582    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-credential-socket\") pod \"9f35ea3d-b5b9-47b2-800d-78cc466867af\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") "
Nov 02 20:29:35.198840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198607    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f35ea3d-b5b9-47b2-800d-78cc466867af-istiod-ca-cert" pod "9f35ea3d-b5b9-47b2-800d-78cc466867af" (UID: "9f35ea3d-b5b9-47b2-800d-78cc466867af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:35.198840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198704    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-data" pod "9f35ea3d-b5b9-47b2-800d-78cc466867af" (UID: "9f35ea3d-b5b9-47b2-800d-78cc466867af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:35.198840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198741    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-socket" pod "9f35ea3d-b5b9-47b2-800d-78cc466867af" (UID: "9f35ea3d-b5b9-47b2-800d-78cc466867af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:35.198840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198622    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-kube-api-access-mbtvt\" (UniqueName: \"kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-kube-api-access-mbtvt\") pod \"9f35ea3d-b5b9-47b2-800d-78cc466867af\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") "
Nov 02 20:29:35.199053 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198899    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-token\" (UniqueName: \"kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-token\") pod \"9f35ea3d-b5b9-47b2-800d-78cc466867af\" (UID: \"9f35ea3d-b5b9-47b2-800d-78cc466867af\") "
Nov 02 20:29:35.199053 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198935    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-certs" pod "9f35ea3d-b5b9-47b2-800d-78cc466867af" (UID: "9f35ea3d-b5b9-47b2-800d-78cc466867af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:35.199053 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.198962    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-credential-socket" pod "9f35ea3d-b5b9-47b2-800d-78cc466867af" (UID: "9f35ea3d-b5b9-47b2-800d-78cc466867af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:35.199397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.199375    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.199485 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.199402    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.199485 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.199417    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.199485 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.199432    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.199485 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.199446    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.199485 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.199459    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.199485 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.199474    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drbhb\" (UniqueName: \"kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-kube-api-access-drbhb\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.199835 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.199490    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f35ea3d-b5b9-47b2-800d-78cc466867af-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.199835 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.199504    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a0e46b-fc81-4841-a517-b56e0a54061d-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.200983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.200948    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-envoy" pod "9f35ea3d-b5b9-47b2-800d-78cc466867af" (UID: "9f35ea3d-b5b9-47b2-800d-78cc466867af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:35.200983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.200958    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-podinfo" pod "9f35ea3d-b5b9-47b2-800d-78cc466867af" (UID: "9f35ea3d-b5b9-47b2-800d-78cc466867af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:35.201167 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.201071    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-token" pod "9f35ea3d-b5b9-47b2-800d-78cc466867af" (UID: "9f35ea3d-b5b9-47b2-800d-78cc466867af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:35.201167 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.201085    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-kube-api-access-mbtvt" pod "9f35ea3d-b5b9-47b2-800d-78cc466867af" (UID: "9f35ea3d-b5b9-47b2-800d-78cc466867af"). InnerVolumeSpecName "kube-api-access-mbtvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:35.244987 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.244639    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:35.300361 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.300327    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.300361 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.300356    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbtvt\" (UniqueName: \"kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-kube-api-access-mbtvt\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.300361 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.300367    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.300683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.300377    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f35ea3d-b5b9-47b2-800d-78cc466867af-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:35.409215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.409179    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"]
Nov 02 20:29:35.412792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.412750    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-phwcd"]
Nov 02 20:29:35.454410 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.454323    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:35.984185 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.984156    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"
Nov 02 20:29:35.984185 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.984176    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7" event={"ID":"9f35ea3d-b5b9-47b2-800d-78cc466867af","Type":"ContainerDied","Data":"c1e0cf610dc73c17e4aec414d43babe9700124d0c10266305d17286de268570f"}
Nov 02 20:29:35.984935 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:35.984252    5224 scope.go:122] "RemoveContainer" containerID="5cf2aa375b80445714c151924b8c4b658ba40ff4006dd8036cd44f3bc4e1b167"
Nov 02 20:29:36.152986 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:36.152947    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"]
Nov 02 20:29:36.203623 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:36.203580    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-wvqv7"]
Nov 02 20:29:36.906758 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:36.906714    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:37.167836 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:37.167740    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a0e46b-fc81-4841-a517-b56e0a54061d" path="/var/lib/kubelet/pods/98a0e46b-fc81-4841-a517-b56e0a54061d/volumes"
Nov 02 20:29:37.168298 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:37.168282    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f35ea3d-b5b9-47b2-800d-78cc466867af" path="/var/lib/kubelet/pods/9f35ea3d-b5b9-47b2-800d-78cc466867af/volumes"
Nov 02 20:29:40.994229 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:40.994183    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:42.123513 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:42.123465    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:42.246327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:42.246288    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"]
Nov 02 20:29:42.246599 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:42.246571    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w" podUID="b0fff547-0622-43a8-8305-300fedb8a311" containerName="istio-proxy" containerID="cri-o://3b19830a004e899b4ca79b7ec3ac8b5fb7b28282980fc94974fe49fb7e5921d0" gracePeriod=30
Nov 02 20:29:46.766043 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766000    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"]
Nov 02 20:29:46.766590 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766570    5224 cpu_manager.go:414] "containerMap: removing container" podUID="98a0e46b-fc81-4841-a517-b56e0a54061d" containerName="istio-proxy"
Nov 02 20:29:46.766640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766593    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a0e46b-fc81-4841-a517-b56e0a54061d" containerName="istio-proxy"
Nov 02 20:29:46.766640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766607    5224 cpu_manager.go:414] "containerMap: removing container" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy"
Nov 02 20:29:46.766640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766612    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy"
Nov 02 20:29:46.766640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766623    5224 cpu_manager.go:414] "containerMap: removing container" podUID="7035711a-f7cb-48f5-a7fc-d66673ea55c2" containerName="payload-processing"
Nov 02 20:29:46.766640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766628    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7035711a-f7cb-48f5-a7fc-d66673ea55c2" containerName="payload-processing"
Nov 02 20:29:46.766809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766644    5224 cpu_manager.go:414] "containerMap: removing container" podUID="65f01ab5-ac30-406c-b8a6-63a76e042836" containerName="main"
Nov 02 20:29:46.766809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766651    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f01ab5-ac30-406c-b8a6-63a76e042836" containerName="main"
Nov 02 20:29:46.766809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766664    5224 cpu_manager.go:414] "containerMap: removing container" podUID="44fc8090-e113-4fdc-a497-20d55f78b354" containerName="payload-pre-processing"
Nov 02 20:29:46.766809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766669    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44fc8090-e113-4fdc-a497-20d55f78b354" containerName="payload-pre-processing"
Nov 02 20:29:46.766809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766677    5224 cpu_manager.go:414] "containerMap: removing container" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy"
Nov 02 20:29:46.766809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766682    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy"
Nov 02 20:29:46.766809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766689    5224 cpu_manager.go:414] "containerMap: removing container" podUID="9f35ea3d-b5b9-47b2-800d-78cc466867af" containerName="istio-proxy"
Nov 02 20:29:46.766809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766693    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f35ea3d-b5b9-47b2-800d-78cc466867af" containerName="istio-proxy"
Nov 02 20:29:46.766809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766791    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="44fc8090-e113-4fdc-a497-20d55f78b354" containerName="payload-pre-processing"
Nov 02 20:29:46.766809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766805    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy"
Nov 02 20:29:46.766809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766813    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f35ea3d-b5b9-47b2-800d-78cc466867af" containerName="istio-proxy"
Nov 02 20:29:46.767180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766823    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a0e46b-fc81-4841-a517-b56e0a54061d" containerName="istio-proxy"
Nov 02 20:29:46.767180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766829    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="8708cdb6-cf04-4f89-b0fc-a0a3f9fb0e0e" containerName="istio-proxy"
Nov 02 20:29:46.767180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766835    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7035711a-f7cb-48f5-a7fc-d66673ea55c2" containerName="payload-processing"
Nov 02 20:29:46.767180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.766844    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f01ab5-ac30-406c-b8a6-63a76e042836" containerName="main"
Nov 02 20:29:46.771723 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.771706    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:46.823264 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.823228    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"]
Nov 02 20:29:46.834382 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.834347    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdxbf\" (UniqueName: \"kubernetes.io/projected/851939a7-2f00-4158-9cfb-9a43641a5c5e-kube-api-access-xdxbf\") pod \"maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:46.834382 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.834390    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-metrics-certs\") pod \"maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:46.834625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.834490    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851939a7-2f00-4158-9cfb-9a43641a5c5e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:46.834625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.834530    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:46.872024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.871988    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-93816f-serving-cert\""
Nov 02 20:29:46.935977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.935943    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxbf\" (UniqueName: \"kubernetes.io/projected/851939a7-2f00-4158-9cfb-9a43641a5c5e-kube-api-access-xdxbf\") pod \"maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:46.935977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.935980    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-metrics-certs\") pod \"maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:46.936215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.936011    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851939a7-2f00-4158-9cfb-9a43641a5c5e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:46.936215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.936032    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:46.936287 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:29:46.936232    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-serving-cert: secret "maas-api-e2e-praxis-a-93816f-serving-cert" not found
Nov 02 20:29:46.936287 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:29:46.936285    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-maas-api-tls podName:851939a7-2f00-4158-9cfb-9a43641a5c5e nodeName:}" failed. No retries permitted until 2026-11-02 20:29:47.436269466 +0000 UTC m=+2468.433958066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-maas-api-tls") pod "maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g" (UID: "851939a7-2f00-4158-9cfb-9a43641a5c5e") : secret "maas-api-e2e-praxis-a-93816f-serving-cert" not found
Nov 02 20:29:46.936674 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.936655    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851939a7-2f00-4158-9cfb-9a43641a5c5e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:46.938467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.938444    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-metrics-certs\") pod \"maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:46.994720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:46.994686    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdxbf\" (UniqueName: \"kubernetes.io/projected/851939a7-2f00-4158-9cfb-9a43641a5c5e-kube-api-access-xdxbf\") pod \"maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:47.007256 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.007219    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:47.228549 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.228441    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:47.407220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.407173    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:47.442872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.442835    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:47.445971 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.445940    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:47.518116 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.518088    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:29:47.544034 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.543996    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-envoy\") pod \"b0fff547-0622-43a8-8305-300fedb8a311\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") "
Nov 02 20:29:47.544221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.544050    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fff547-0622-43a8-8305-300fedb8a311-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fff547-0622-43a8-8305-300fedb8a311-istiod-ca-cert\") pod \"b0fff547-0622-43a8-8305-300fedb8a311\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") "
Nov 02 20:29:47.544221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.544084    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-istio-token\" (UniqueName: \"kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-istio-token\") pod \"b0fff547-0622-43a8-8305-300fedb8a311\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") "
Nov 02 20:29:47.544221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.544151    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-credential-socket\") pod \"b0fff547-0622-43a8-8305-300fedb8a311\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") "
Nov 02 20:29:47.544417 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.544324    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-data\") pod \"b0fff547-0622-43a8-8305-300fedb8a311\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") "
Nov 02 20:29:47.544417 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.544406    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-kube-api-access-m7gz6\" (UniqueName: \"kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-kube-api-access-m7gz6\") pod \"b0fff547-0622-43a8-8305-300fedb8a311\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") "
Nov 02 20:29:47.544522 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.544465    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-credential-socket" pod "b0fff547-0622-43a8-8305-300fedb8a311" (UID: "b0fff547-0622-43a8-8305-300fedb8a311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:47.544522 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.544491    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-socket\") pod \"b0fff547-0622-43a8-8305-300fedb8a311\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") "
Nov 02 20:29:47.544639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.544564    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0fff547-0622-43a8-8305-300fedb8a311-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fff547-0622-43a8-8305-300fedb8a311-istio-podinfo\") pod \"b0fff547-0622-43a8-8305-300fedb8a311\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") "
Nov 02 20:29:47.544639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.544568    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fff547-0622-43a8-8305-300fedb8a311-istiod-ca-cert" pod "b0fff547-0622-43a8-8305-300fedb8a311" (UID: "b0fff547-0622-43a8-8305-300fedb8a311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:47.544639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.544624    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-certs\") pod \"b0fff547-0622-43a8-8305-300fedb8a311\" (UID: \"b0fff547-0622-43a8-8305-300fedb8a311\") "
Nov 02 20:29:47.544639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.544627    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-data" pod "b0fff547-0622-43a8-8305-300fedb8a311" (UID: "b0fff547-0622-43a8-8305-300fedb8a311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:47.544881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.544863    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-socket" pod "b0fff547-0622-43a8-8305-300fedb8a311" (UID: "b0fff547-0622-43a8-8305-300fedb8a311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:47.545189 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.545165    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fff547-0622-43a8-8305-300fedb8a311-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:47.545189 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.545190    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:47.545423 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.545204    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:47.545423 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.545201    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-certs" pod "b0fff547-0622-43a8-8305-300fedb8a311" (UID: "b0fff547-0622-43a8-8305-300fedb8a311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:47.545423 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.545216    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:47.546564 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.546538    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-istio-token" pod "b0fff547-0622-43a8-8305-300fedb8a311" (UID: "b0fff547-0622-43a8-8305-300fedb8a311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:47.546704 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.546623    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-kube-api-access-m7gz6" pod "b0fff547-0622-43a8-8305-300fedb8a311" (UID: "b0fff547-0622-43a8-8305-300fedb8a311"). InnerVolumeSpecName "kube-api-access-m7gz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:47.546821 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.546806    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-envoy" pod "b0fff547-0622-43a8-8305-300fedb8a311" (UID: "b0fff547-0622-43a8-8305-300fedb8a311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:47.546951 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.546936    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0fff547-0622-43a8-8305-300fedb8a311-istio-podinfo" pod "b0fff547-0622-43a8-8305-300fedb8a311" (UID: "b0fff547-0622-43a8-8305-300fedb8a311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:47.646915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.646878    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7gz6\" (UniqueName: \"kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-kube-api-access-m7gz6\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:47.646915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.646911    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fff547-0622-43a8-8305-300fedb8a311-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:47.646915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.646921    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:47.647273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.646930    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fff547-0622-43a8-8305-300fedb8a311-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:47.647273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.646940    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0fff547-0622-43a8-8305-300fedb8a311-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:47.685649 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:47.685611    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:48.057383 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:29:48.057343    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851939a7_2f00_4158_9cfb_9a43641a5c5e.slice/crio-0f38c6d44f5dcc3921c6ba46214f0b75c53af0f77c967ba9667107691af1fc0d WatchSource:0}: Error finding container 0f38c6d44f5dcc3921c6ba46214f0b75c53af0f77c967ba9667107691af1fc0d: Status 404 returned error can't find the container with id 0f38c6d44f5dcc3921c6ba46214f0b75c53af0f77c967ba9667107691af1fc0d
Nov 02 20:29:48.060268 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:48.060239    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"]
Nov 02 20:29:48.060987 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:48.060952    5224 generic.go:356] "Generic (PLEG): container finished" podID="b0fff547-0622-43a8-8305-300fedb8a311" containerID="3b19830a004e899b4ca79b7ec3ac8b5fb7b28282980fc94974fe49fb7e5921d0" exitCode=0
Nov 02 20:29:48.061084 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:48.061005    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"
Nov 02 20:29:48.061135 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:48.060999    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w" event={"ID":"b0fff547-0622-43a8-8305-300fedb8a311","Type":"ContainerDied","Data":"3b19830a004e899b4ca79b7ec3ac8b5fb7b28282980fc94974fe49fb7e5921d0"}
Nov 02 20:29:48.061135 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:48.061122    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w" event={"ID":"b0fff547-0622-43a8-8305-300fedb8a311","Type":"ContainerDied","Data":"ec2e2310a987ae3cc9ab9335f786a2f920c5e398ec2dcb66688dc2d42c8cf629"}
Nov 02 20:29:48.061218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:48.061151    5224 scope.go:122] "RemoveContainer" containerID="3b19830a004e899b4ca79b7ec3ac8b5fb7b28282980fc94974fe49fb7e5921d0"
Nov 02 20:29:48.071953 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:48.071932    5224 scope.go:122] "RemoveContainer" containerID="3b19830a004e899b4ca79b7ec3ac8b5fb7b28282980fc94974fe49fb7e5921d0"
Nov 02 20:29:48.072254 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:29:48.072232    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b19830a004e899b4ca79b7ec3ac8b5fb7b28282980fc94974fe49fb7e5921d0\": container with ID starting with 3b19830a004e899b4ca79b7ec3ac8b5fb7b28282980fc94974fe49fb7e5921d0 not found: ID does not exist" containerID="3b19830a004e899b4ca79b7ec3ac8b5fb7b28282980fc94974fe49fb7e5921d0"
Nov 02 20:29:48.072289 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:48.072266    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b19830a004e899b4ca79b7ec3ac8b5fb7b28282980fc94974fe49fb7e5921d0"} err="failed to get container status \"3b19830a004e899b4ca79b7ec3ac8b5fb7b28282980fc94974fe49fb7e5921d0\": rpc error: code = NotFound desc = could not find container \"3b19830a004e899b4ca79b7ec3ac8b5fb7b28282980fc94974fe49fb7e5921d0\": container with ID starting with 3b19830a004e899b4ca79b7ec3ac8b5fb7b28282980fc94974fe49fb7e5921d0 not found: ID does not exist"
Nov 02 20:29:48.231320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:48.231226    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"]
Nov 02 20:29:48.282297 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:48.282251    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7q55w"]
Nov 02 20:29:49.070549 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:49.070512    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g" event={"ID":"851939a7-2f00-4158-9cfb-9a43641a5c5e","Type":"ContainerStarted","Data":"0f38c6d44f5dcc3921c6ba46214f0b75c53af0f77c967ba9667107691af1fc0d"}
Nov 02 20:29:49.167800 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:49.167734    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0fff547-0622-43a8-8305-300fedb8a311" path="/var/lib/kubelet/pods/b0fff547-0622-43a8-8305-300fedb8a311/volumes"
Nov 02 20:29:50.078738 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:50.078698    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g" event={"ID":"851939a7-2f00-4158-9cfb-9a43641a5c5e","Type":"ContainerStarted","Data":"2b38805d0487dadef5e381a108480079cceacc30c71ca99e3cfb196a09e67c0e"}
Nov 02 20:29:50.079279 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:50.078814    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:50.195494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:50.195431    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g" podStartSLOduration=3.26434921 podStartE2EDuration="4.195414861s" totalImagesPullingTime="931.065651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:50.194984626 +0000 UTC m=+2471.192673275" watchObservedRunningTime="2026-11-02 20:29:50.195414861 +0000 UTC m=+2471.193103470"
Nov 02 20:29:51.073511 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:51.073470    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:29:51.467396 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:51.467296    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:53.261674 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:53.261617    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:56.084803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:56.084748    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:56.090059 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:56.090022    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:29:56.358090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:56.358006    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:57.359942 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:29:57.359908    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:29:57.732165 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:57.732068    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:58.260236 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:58.260196    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:58.439019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:58.438977    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:58.623922 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:58.623874    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:59.066942 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:59.066903    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:29:59.378842 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:29:59.378687    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:00.168380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.168337    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"]
Nov 02 20:30:00.169079 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.169055    5224 cpu_manager.go:414] "containerMap: removing container" podUID="b0fff547-0622-43a8-8305-300fedb8a311" containerName="istio-proxy"
Nov 02 20:30:00.169181 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.169080    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fff547-0622-43a8-8305-300fedb8a311" containerName="istio-proxy"
Nov 02 20:30:00.169238 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.169197    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fff547-0622-43a8-8305-300fedb8a311" containerName="istio-proxy"
Nov 02 20:30:00.173951 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.173926    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"
Nov 02 20:30:00.180206 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.180177    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"]
Nov 02 20:30:00.183838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.183819    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"]
Nov 02 20:30:00.183968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.183948    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"
Nov 02 20:30:00.187401 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.187376    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"
Nov 02 20:30:00.219852 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.219820    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"]
Nov 02 20:30:00.224161 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.224141    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"]
Nov 02 20:30:00.224266 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.224250    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"
Nov 02 20:30:00.228725 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.228702    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"
Nov 02 20:30:00.229464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.229443    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:30:00.260321 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.260288    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"]
Nov 02 20:30:00.289391 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.289350    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xlrb\" (UniqueName: \"kubernetes.io/projected/0c254164-9b88-49e9-9e90-575daf8b855e-kube-api-access-5xlrb\") pod \"maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr\" (UID: \"0c254164-9b88-49e9-9e90-575daf8b855e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"
Nov 02 20:30:00.289575 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.289476    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ww9c\" (UniqueName: \"kubernetes.io/projected/3934517c-ae41-4e8e-9651-dda812f7daea-kube-api-access-4ww9c\") pod \"maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr\" (UID: \"3934517c-ae41-4e8e-9651-dda812f7daea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"
Nov 02 20:30:00.289575 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.289553    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c254164-9b88-49e9-9e90-575daf8b855e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr\" (UID: \"0c254164-9b88-49e9-9e90-575daf8b855e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"
Nov 02 20:30:00.289705 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.289681    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d45mv\" (UniqueName: \"kubernetes.io/projected/201c4e04-0153-4013-978c-2d8be535a640-kube-api-access-d45mv\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8\" (UID: \"201c4e04-0153-4013-978c-2d8be535a640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"
Nov 02 20:30:00.289760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.289723    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/201c4e04-0153-4013-978c-2d8be535a640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8\" (UID: \"201c4e04-0153-4013-978c-2d8be535a640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"
Nov 02 20:30:00.289851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.289834    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c75ab44-108b-4a6c-8a99-8ee871f8894a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h\" (UID: \"1c75ab44-108b-4a6c-8a99-8ee871f8894a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"
Nov 02 20:30:00.289912 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.289874    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3934517c-ae41-4e8e-9651-dda812f7daea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr\" (UID: \"3934517c-ae41-4e8e-9651-dda812f7daea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"
Nov 02 20:30:00.289961 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.289926    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068b0dec-2759-4b13-ab24-090ee50010de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm\" (UID: \"068b0dec-2759-4b13-ab24-090ee50010de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"
Nov 02 20:30:00.290014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.289968    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lb8z\" (UniqueName: \"kubernetes.io/projected/068b0dec-2759-4b13-ab24-090ee50010de-kube-api-access-7lb8z\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm\" (UID: \"068b0dec-2759-4b13-ab24-090ee50010de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"
Nov 02 20:30:00.290014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.289990    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znt5h\" (UniqueName: \"kubernetes.io/projected/1c75ab44-108b-4a6c-8a99-8ee871f8894a-kube-api-access-znt5h\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h\" (UID: \"1c75ab44-108b-4a6c-8a99-8ee871f8894a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"
Nov 02 20:30:00.303112 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.303078    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"]
Nov 02 20:30:00.306061 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.306028    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"]
Nov 02 20:30:00.308639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.308611    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"]
Nov 02 20:30:00.351031 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.350992    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"]
Nov 02 20:30:00.391259 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.391225    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c254164-9b88-49e9-9e90-575daf8b855e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr\" (UID: \"0c254164-9b88-49e9-9e90-575daf8b855e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"
Nov 02 20:30:00.391487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.391319    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d45mv\" (UniqueName: \"kubernetes.io/projected/201c4e04-0153-4013-978c-2d8be535a640-kube-api-access-d45mv\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8\" (UID: \"201c4e04-0153-4013-978c-2d8be535a640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"
Nov 02 20:30:00.391487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.391350    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/201c4e04-0153-4013-978c-2d8be535a640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8\" (UID: \"201c4e04-0153-4013-978c-2d8be535a640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"
Nov 02 20:30:00.391487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.391411    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c75ab44-108b-4a6c-8a99-8ee871f8894a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h\" (UID: \"1c75ab44-108b-4a6c-8a99-8ee871f8894a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"
Nov 02 20:30:00.391487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.391457    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3934517c-ae41-4e8e-9651-dda812f7daea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr\" (UID: \"3934517c-ae41-4e8e-9651-dda812f7daea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"
Nov 02 20:30:00.391741 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.391496    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068b0dec-2759-4b13-ab24-090ee50010de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm\" (UID: \"068b0dec-2759-4b13-ab24-090ee50010de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"
Nov 02 20:30:00.391741 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.391525    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lb8z\" (UniqueName: \"kubernetes.io/projected/068b0dec-2759-4b13-ab24-090ee50010de-kube-api-access-7lb8z\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm\" (UID: \"068b0dec-2759-4b13-ab24-090ee50010de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"
Nov 02 20:30:00.391741 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.391557    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znt5h\" (UniqueName: \"kubernetes.io/projected/1c75ab44-108b-4a6c-8a99-8ee871f8894a-kube-api-access-znt5h\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h\" (UID: \"1c75ab44-108b-4a6c-8a99-8ee871f8894a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"
Nov 02 20:30:00.391741 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.391614    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xlrb\" (UniqueName: \"kubernetes.io/projected/0c254164-9b88-49e9-9e90-575daf8b855e-kube-api-access-5xlrb\") pod \"maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr\" (UID: \"0c254164-9b88-49e9-9e90-575daf8b855e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"
Nov 02 20:30:00.391741 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.391686    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ww9c\" (UniqueName: \"kubernetes.io/projected/3934517c-ae41-4e8e-9651-dda812f7daea-kube-api-access-4ww9c\") pod \"maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr\" (UID: \"3934517c-ae41-4e8e-9651-dda812f7daea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"
Nov 02 20:30:00.392148 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.392127    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c254164-9b88-49e9-9e90-575daf8b855e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr\" (UID: \"0c254164-9b88-49e9-9e90-575daf8b855e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"
Nov 02 20:30:00.392275 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.392250    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3934517c-ae41-4e8e-9651-dda812f7daea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr\" (UID: \"3934517c-ae41-4e8e-9651-dda812f7daea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"
Nov 02 20:30:00.392313 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.392250    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c75ab44-108b-4a6c-8a99-8ee871f8894a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h\" (UID: \"1c75ab44-108b-4a6c-8a99-8ee871f8894a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"
Nov 02 20:30:00.392313 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.392297    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068b0dec-2759-4b13-ab24-090ee50010de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm\" (UID: \"068b0dec-2759-4b13-ab24-090ee50010de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"
Nov 02 20:30:00.392488 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.392472    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/201c4e04-0153-4013-978c-2d8be535a640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8\" (UID: \"201c4e04-0153-4013-978c-2d8be535a640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"
Nov 02 20:30:00.449630 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.449592    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xlrb\" (UniqueName: \"kubernetes.io/projected/0c254164-9b88-49e9-9e90-575daf8b855e-kube-api-access-5xlrb\") pod \"maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr\" (UID: \"0c254164-9b88-49e9-9e90-575daf8b855e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"
Nov 02 20:30:00.449839 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.449758    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d45mv\" (UniqueName: \"kubernetes.io/projected/201c4e04-0153-4013-978c-2d8be535a640-kube-api-access-d45mv\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8\" (UID: \"201c4e04-0153-4013-978c-2d8be535a640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"
Nov 02 20:30:00.449955 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.449925    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znt5h\" (UniqueName: \"kubernetes.io/projected/1c75ab44-108b-4a6c-8a99-8ee871f8894a-kube-api-access-znt5h\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h\" (UID: \"1c75ab44-108b-4a6c-8a99-8ee871f8894a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"
Nov 02 20:30:00.450551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.450527    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ww9c\" (UniqueName: \"kubernetes.io/projected/3934517c-ae41-4e8e-9651-dda812f7daea-kube-api-access-4ww9c\") pod \"maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr\" (UID: \"3934517c-ae41-4e8e-9651-dda812f7daea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"
Nov 02 20:30:00.450683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.450667    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lb8z\" (UniqueName: \"kubernetes.io/projected/068b0dec-2759-4b13-ab24-090ee50010de-kube-api-access-7lb8z\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm\" (UID: \"068b0dec-2759-4b13-ab24-090ee50010de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"
Nov 02 20:30:00.491223 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.491170    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"
Nov 02 20:30:00.505383 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.505344    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"
Nov 02 20:30:00.512346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.512313    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"
Nov 02 20:30:00.540406 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.540370    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"
Nov 02 20:30:00.548604 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.548567    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"
Nov 02 20:30:00.868203 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:30:00.868173    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3934517c_ae41_4e8e_9651_dda812f7daea.slice/crio-27ba4a7b807194ec35eb5d40ed9d5c01dacee39d96851e5c52b58b56766d71b6 WatchSource:0}: Error finding container 27ba4a7b807194ec35eb5d40ed9d5c01dacee39d96851e5c52b58b56766d71b6: Status 404 returned error can't find the container with id 27ba4a7b807194ec35eb5d40ed9d5c01dacee39d96851e5c52b58b56766d71b6
Nov 02 20:30:00.871422 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.871396    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"]
Nov 02 20:30:00.913733 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:30:00.913703    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068b0dec_2759_4b13_ab24_090ee50010de.slice/crio-2e9bd6e36f2dccc9e532ad9eaed3bd66c057b9a3653f0dc0f207c6e24c39c6ba WatchSource:0}: Error finding container 2e9bd6e36f2dccc9e532ad9eaed3bd66c057b9a3653f0dc0f207c6e24c39c6ba: Status 404 returned error can't find the container with id 2e9bd6e36f2dccc9e532ad9eaed3bd66c057b9a3653f0dc0f207c6e24c39c6ba
Nov 02 20:30:00.915729 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:30:00.915695    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c75ab44_108b_4a6c_8a99_8ee871f8894a.slice/crio-0e8b0d2ffb959c8d675a8c041a40f7c193ea124714c6c84ea9bbcd78936bb406 WatchSource:0}: Error finding container 0e8b0d2ffb959c8d675a8c041a40f7c193ea124714c6c84ea9bbcd78936bb406: Status 404 returned error can't find the container with id 0e8b0d2ffb959c8d675a8c041a40f7c193ea124714c6c84ea9bbcd78936bb406
Nov 02 20:30:00.916795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.916740    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"]
Nov 02 20:30:00.919409 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.919384    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"]
Nov 02 20:30:00.971232 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:30:00.971189    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c254164_9b88_49e9_9e90_575daf8b855e.slice/crio-c35072ce429458b6fc72a70b749c8f683d8beb3832de1b716392373a41b2b585 WatchSource:0}: Error finding container c35072ce429458b6fc72a70b749c8f683d8beb3832de1b716392373a41b2b585: Status 404 returned error can't find the container with id c35072ce429458b6fc72a70b749c8f683d8beb3832de1b716392373a41b2b585
Nov 02 20:30:00.974893 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.974866    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"]
Nov 02 20:30:00.977934 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:30:00.977896    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201c4e04_0153_4013_978c_2d8be535a640.slice/crio-efcbde59ca211dfde9b40490c586ae0e9f73443f7c785dd6346dfa32b64895f5 WatchSource:0}: Error finding container efcbde59ca211dfde9b40490c586ae0e9f73443f7c785dd6346dfa32b64895f5: Status 404 returned error can't find the container with id efcbde59ca211dfde9b40490c586ae0e9f73443f7c785dd6346dfa32b64895f5
Nov 02 20:30:00.980112 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:00.980092    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"]
Nov 02 20:30:01.145607 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:01.145571    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8" event={"ID":"201c4e04-0153-4013-978c-2d8be535a640","Type":"ContainerStarted","Data":"efcbde59ca211dfde9b40490c586ae0e9f73443f7c785dd6346dfa32b64895f5"}
Nov 02 20:30:01.146894 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:01.146859    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm" event={"ID":"068b0dec-2759-4b13-ab24-090ee50010de","Type":"ContainerStarted","Data":"2e9bd6e36f2dccc9e532ad9eaed3bd66c057b9a3653f0dc0f207c6e24c39c6ba"}
Nov 02 20:30:01.148092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:01.148059    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr" event={"ID":"3934517c-ae41-4e8e-9651-dda812f7daea","Type":"ContainerStarted","Data":"27ba4a7b807194ec35eb5d40ed9d5c01dacee39d96851e5c52b58b56766d71b6"}
Nov 02 20:30:01.149863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:01.149829    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h" event={"ID":"1c75ab44-108b-4a6c-8a99-8ee871f8894a","Type":"ContainerStarted","Data":"0e8b0d2ffb959c8d675a8c041a40f7c193ea124714c6c84ea9bbcd78936bb406"}
Nov 02 20:30:01.151254 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:01.151233    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr" event={"ID":"0c254164-9b88-49e9-9e90-575daf8b855e","Type":"ContainerStarted","Data":"c35072ce429458b6fc72a70b749c8f683d8beb3832de1b716392373a41b2b585"}
Nov 02 20:30:02.160081 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.160046    5224 generic.go:356] "Generic (PLEG): container finished" podID="201c4e04-0153-4013-978c-2d8be535a640" containerID="9c5d5a5262166a98830bf1964d1aa63ac620887a4ff38e70605fac95c34abf82" exitCode=60
Nov 02 20:30:02.160531 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.160128    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8" event={"ID":"201c4e04-0153-4013-978c-2d8be535a640","Type":"ContainerDied","Data":"9c5d5a5262166a98830bf1964d1aa63ac620887a4ff38e70605fac95c34abf82"}
Nov 02 20:30:02.160531 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.160404    5224 scope.go:122] "RemoveContainer" containerID="9c5d5a5262166a98830bf1964d1aa63ac620887a4ff38e70605fac95c34abf82"
Nov 02 20:30:02.161923 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.161896    5224 generic.go:356] "Generic (PLEG): container finished" podID="068b0dec-2759-4b13-ab24-090ee50010de" containerID="b0676693bc02cd35ab44875c605d640194110441ca9bf369887d01ee05b0b286" exitCode=60
Nov 02 20:30:02.162066 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.161973    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm" event={"ID":"068b0dec-2759-4b13-ab24-090ee50010de","Type":"ContainerDied","Data":"b0676693bc02cd35ab44875c605d640194110441ca9bf369887d01ee05b0b286"}
Nov 02 20:30:02.162301 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.162286    5224 scope.go:122] "RemoveContainer" containerID="b0676693bc02cd35ab44875c605d640194110441ca9bf369887d01ee05b0b286"
Nov 02 20:30:02.163828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.163811    5224 generic.go:356] "Generic (PLEG): container finished" podID="3934517c-ae41-4e8e-9651-dda812f7daea" containerID="3bab5a9cae3078d4f0fe84953802db95717fa3f2910104a8eb40f04df8ef0b20" exitCode=60
Nov 02 20:30:02.163924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.163901    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr" event={"ID":"3934517c-ae41-4e8e-9651-dda812f7daea","Type":"ContainerDied","Data":"3bab5a9cae3078d4f0fe84953802db95717fa3f2910104a8eb40f04df8ef0b20"}
Nov 02 20:30:02.164148 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.164133    5224 scope.go:122] "RemoveContainer" containerID="3bab5a9cae3078d4f0fe84953802db95717fa3f2910104a8eb40f04df8ef0b20"
Nov 02 20:30:02.165634 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.165614    5224 generic.go:356] "Generic (PLEG): container finished" podID="1c75ab44-108b-4a6c-8a99-8ee871f8894a" containerID="186fd8f92902dadbc854b8d10caff78378afd9e6553d447a1b34ec996ef972f7" exitCode=60
Nov 02 20:30:02.165709 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.165688    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h" event={"ID":"1c75ab44-108b-4a6c-8a99-8ee871f8894a","Type":"ContainerDied","Data":"186fd8f92902dadbc854b8d10caff78378afd9e6553d447a1b34ec996ef972f7"}
Nov 02 20:30:02.165896 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.165882    5224 scope.go:122] "RemoveContainer" containerID="186fd8f92902dadbc854b8d10caff78378afd9e6553d447a1b34ec996ef972f7"
Nov 02 20:30:02.167313 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.167295    5224 generic.go:356] "Generic (PLEG): container finished" podID="0c254164-9b88-49e9-9e90-575daf8b855e" containerID="d4d1f0a966c2a98690a9c4b7ebe28db18f9c9ce6dbbbe73f69c2960e3bedba85" exitCode=60
Nov 02 20:30:02.167425 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.167384    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr" event={"ID":"0c254164-9b88-49e9-9e90-575daf8b855e","Type":"ContainerDied","Data":"d4d1f0a966c2a98690a9c4b7ebe28db18f9c9ce6dbbbe73f69c2960e3bedba85"}
Nov 02 20:30:02.167571 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:02.167555    5224 scope.go:122] "RemoveContainer" containerID="d4d1f0a966c2a98690a9c4b7ebe28db18f9c9ce6dbbbe73f69c2960e3bedba85"
Nov 02 20:30:03.176868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.176829    5224 generic.go:356] "Generic (PLEG): container finished" podID="201c4e04-0153-4013-978c-2d8be535a640" containerID="3c82ec334f30bb724509c1423714a63fb014d7656864a62ea17ec7e9f8910a26" exitCode=60
Nov 02 20:30:03.177342 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.176916    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8" event={"ID":"201c4e04-0153-4013-978c-2d8be535a640","Type":"ContainerDied","Data":"3c82ec334f30bb724509c1423714a63fb014d7656864a62ea17ec7e9f8910a26"}
Nov 02 20:30:03.177342 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.176983    5224 scope.go:122] "RemoveContainer" containerID="9c5d5a5262166a98830bf1964d1aa63ac620887a4ff38e70605fac95c34abf82"
Nov 02 20:30:03.177342 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.177250    5224 scope.go:122] "RemoveContainer" containerID="3c82ec334f30bb724509c1423714a63fb014d7656864a62ea17ec7e9f8910a26"
Nov 02 20:30:03.177664 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:03.177589    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8_odh-ai-gateway-infra(201c4e04-0153-4013-978c-2d8be535a640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8" podUID="201c4e04-0153-4013-978c-2d8be535a640"
Nov 02 20:30:03.179269 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.179236    5224 generic.go:356] "Generic (PLEG): container finished" podID="068b0dec-2759-4b13-ab24-090ee50010de" containerID="d3de42d6d667f4924dc2c7a569abe11ec4cc3354e1ebdf6b81cfad03b9fa3bbf" exitCode=60
Nov 02 20:30:03.179405 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.179315    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm" event={"ID":"068b0dec-2759-4b13-ab24-090ee50010de","Type":"ContainerDied","Data":"d3de42d6d667f4924dc2c7a569abe11ec4cc3354e1ebdf6b81cfad03b9fa3bbf"}
Nov 02 20:30:03.179607 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.179588    5224 scope.go:122] "RemoveContainer" containerID="d3de42d6d667f4924dc2c7a569abe11ec4cc3354e1ebdf6b81cfad03b9fa3bbf"
Nov 02 20:30:03.179968 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:03.179945    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm_odh-ai-gateway-infra(068b0dec-2759-4b13-ab24-090ee50010de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm" podUID="068b0dec-2759-4b13-ab24-090ee50010de"
Nov 02 20:30:03.181571 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.181549    5224 generic.go:356] "Generic (PLEG): container finished" podID="3934517c-ae41-4e8e-9651-dda812f7daea" containerID="45f669164d9affe785ab7fa95cf283864c290eaaea2aca938fc79e1502f53909" exitCode=60
Nov 02 20:30:03.181662 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.181641    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr" event={"ID":"3934517c-ae41-4e8e-9651-dda812f7daea","Type":"ContainerDied","Data":"45f669164d9affe785ab7fa95cf283864c290eaaea2aca938fc79e1502f53909"}
Nov 02 20:30:03.181953 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.181937    5224 scope.go:122] "RemoveContainer" containerID="45f669164d9affe785ab7fa95cf283864c290eaaea2aca938fc79e1502f53909"
Nov 02 20:30:03.182332 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:03.182300    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr_odh-ai-gateway-infra(3934517c-ae41-4e8e-9651-dda812f7daea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr" podUID="3934517c-ae41-4e8e-9651-dda812f7daea"
Nov 02 20:30:03.183853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.183744    5224 generic.go:356] "Generic (PLEG): container finished" podID="1c75ab44-108b-4a6c-8a99-8ee871f8894a" containerID="78f09992e913c364c60bf14493c783e10bac2a026cbcf7ddb40893d3d03ae31d" exitCode=60
Nov 02 20:30:03.183853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.183815    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h" event={"ID":"1c75ab44-108b-4a6c-8a99-8ee871f8894a","Type":"ContainerDied","Data":"78f09992e913c364c60bf14493c783e10bac2a026cbcf7ddb40893d3d03ae31d"}
Nov 02 20:30:03.184087 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.184074    5224 scope.go:122] "RemoveContainer" containerID="78f09992e913c364c60bf14493c783e10bac2a026cbcf7ddb40893d3d03ae31d"
Nov 02 20:30:03.184330 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:03.184307    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h_odh-ai-gateway-infra(1c75ab44-108b-4a6c-8a99-8ee871f8894a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h" podUID="1c75ab44-108b-4a6c-8a99-8ee871f8894a"
Nov 02 20:30:03.185887 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.185861    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr" event={"ID":"0c254164-9b88-49e9-9e90-575daf8b855e","Type":"ContainerStarted","Data":"0143c8462cde35d63d9033fde1cec668aeab02468ef6a119f1ee554c6cf44269"}
Nov 02 20:30:03.358137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.358111    5224 scope.go:122] "RemoveContainer" containerID="b0676693bc02cd35ab44875c605d640194110441ca9bf369887d01ee05b0b286"
Nov 02 20:30:03.371455 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.371431    5224 scope.go:122] "RemoveContainer" containerID="3bab5a9cae3078d4f0fe84953802db95717fa3f2910104a8eb40f04df8ef0b20"
Nov 02 20:30:03.384994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.384965    5224 scope.go:122] "RemoveContainer" containerID="186fd8f92902dadbc854b8d10caff78378afd9e6553d447a1b34ec996ef972f7"
Nov 02 20:30:03.693230 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.693141    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:03.763986 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:03.763929    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr" podStartSLOduration=3.763912587 podStartE2EDuration="3.763912587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:03.763615768 +0000 UTC m=+2484.761304377" watchObservedRunningTime="2026-11-02 20:30:03.763912587 +0000 UTC m=+2484.761601251"
Nov 02 20:30:04.194944 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:04.194912    5224 generic.go:356] "Generic (PLEG): container finished" podID="0c254164-9b88-49e9-9e90-575daf8b855e" containerID="0143c8462cde35d63d9033fde1cec668aeab02468ef6a119f1ee554c6cf44269" exitCode=60
Nov 02 20:30:04.195420 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:04.195004    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr" event={"ID":"0c254164-9b88-49e9-9e90-575daf8b855e","Type":"ContainerDied","Data":"0143c8462cde35d63d9033fde1cec668aeab02468ef6a119f1ee554c6cf44269"}
Nov 02 20:30:04.195420 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:04.195095    5224 scope.go:122] "RemoveContainer" containerID="d4d1f0a966c2a98690a9c4b7ebe28db18f9c9ce6dbbbe73f69c2960e3bedba85"
Nov 02 20:30:04.195420 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:04.195306    5224 scope.go:122] "RemoveContainer" containerID="0143c8462cde35d63d9033fde1cec668aeab02468ef6a119f1ee554c6cf44269"
Nov 02 20:30:04.195683 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:04.195652    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr_odh-ai-gateway-infra(0c254164-9b88-49e9-9e90-575daf8b855e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr" podUID="0c254164-9b88-49e9-9e90-575daf8b855e"
Nov 02 20:30:04.197462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:04.197443    5224 scope.go:122] "RemoveContainer" containerID="3c82ec334f30bb724509c1423714a63fb014d7656864a62ea17ec7e9f8910a26"
Nov 02 20:30:04.197801 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:04.197760    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8_odh-ai-gateway-infra(201c4e04-0153-4013-978c-2d8be535a640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8" podUID="201c4e04-0153-4013-978c-2d8be535a640"
Nov 02 20:30:04.200071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:04.200043    5224 scope.go:122] "RemoveContainer" containerID="d3de42d6d667f4924dc2c7a569abe11ec4cc3354e1ebdf6b81cfad03b9fa3bbf"
Nov 02 20:30:04.200396 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:04.200367    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm_odh-ai-gateway-infra(068b0dec-2759-4b13-ab24-090ee50010de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm" podUID="068b0dec-2759-4b13-ab24-090ee50010de"
Nov 02 20:30:04.201675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:04.201652    5224 scope.go:122] "RemoveContainer" containerID="45f669164d9affe785ab7fa95cf283864c290eaaea2aca938fc79e1502f53909"
Nov 02 20:30:04.201999 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:04.201969    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr_odh-ai-gateway-infra(3934517c-ae41-4e8e-9651-dda812f7daea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr" podUID="3934517c-ae41-4e8e-9651-dda812f7daea"
Nov 02 20:30:04.203322 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:04.203303    5224 scope.go:122] "RemoveContainer" containerID="78f09992e913c364c60bf14493c783e10bac2a026cbcf7ddb40893d3d03ae31d"
Nov 02 20:30:04.203555 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:04.203538    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h_odh-ai-gateway-infra(1c75ab44-108b-4a6c-8a99-8ee871f8894a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h" podUID="1c75ab44-108b-4a6c-8a99-8ee871f8894a"
Nov 02 20:30:05.115892 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:05.115855    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:05.210896 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:05.210864    5224 scope.go:122] "RemoveContainer" containerID="0143c8462cde35d63d9033fde1cec668aeab02468ef6a119f1ee554c6cf44269"
Nov 02 20:30:05.211307 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:05.211139    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr_odh-ai-gateway-infra(0c254164-9b88-49e9-9e90-575daf8b855e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr" podUID="0c254164-9b88-49e9-9e90-575daf8b855e"
Nov 02 20:30:05.890616 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:05.890579    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"]
Nov 02 20:30:05.890851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:05.890806    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" podUID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" containerName="istio-proxy" containerID="cri-o://d63cecc41ad759491c5e5533bb495d3f39518fc6ffbd5f2cab6e5dbe9822cd1e" gracePeriod=30
Nov 02 20:30:06.630856 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:06.630813    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:06.802452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:06.802409    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:08.991668 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:08.991623    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:09.212437 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:09.212398    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:10.831304 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:10.831261    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:11.169501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.169471    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 20:30:11.247442 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.247403    5224 generic.go:356] "Generic (PLEG): container finished" podID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" containerID="d63cecc41ad759491c5e5533bb495d3f39518fc6ffbd5f2cab6e5dbe9822cd1e" exitCode=0
Nov 02 20:30:11.247604 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.247452    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"
Nov 02 20:30:11.247604 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.247468    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" event={"ID":"8c802b4b-1dec-400b-a1e7-5507933ef8e3","Type":"ContainerDied","Data":"d63cecc41ad759491c5e5533bb495d3f39518fc6ffbd5f2cab6e5dbe9822cd1e"}
Nov 02 20:30:11.247604 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.247517    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z" event={"ID":"8c802b4b-1dec-400b-a1e7-5507933ef8e3","Type":"ContainerDied","Data":"c6122f9f3cc39897b0f2e7fdb1be3ebe35ac66847fe08c9a0650f7d5e9c4335c"}
Nov 02 20:30:11.247604 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.247540    5224 scope.go:122] "RemoveContainer" containerID="d63cecc41ad759491c5e5533bb495d3f39518fc6ffbd5f2cab6e5dbe9822cd1e"
Nov 02 20:30:11.259703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.259680    5224 scope.go:122] "RemoveContainer" containerID="d63cecc41ad759491c5e5533bb495d3f39518fc6ffbd5f2cab6e5dbe9822cd1e"
Nov 02 20:30:11.260072 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:11.260049    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63cecc41ad759491c5e5533bb495d3f39518fc6ffbd5f2cab6e5dbe9822cd1e\": container with ID starting with d63cecc41ad759491c5e5533bb495d3f39518fc6ffbd5f2cab6e5dbe9822cd1e not found: ID does not exist" containerID="d63cecc41ad759491c5e5533bb495d3f39518fc6ffbd5f2cab6e5dbe9822cd1e"
Nov 02 20:30:11.260142 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.260087    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63cecc41ad759491c5e5533bb495d3f39518fc6ffbd5f2cab6e5dbe9822cd1e"} err="failed to get container status \"d63cecc41ad759491c5e5533bb495d3f39518fc6ffbd5f2cab6e5dbe9822cd1e\": rpc error: code = NotFound desc = could not find container \"d63cecc41ad759491c5e5533bb495d3f39518fc6ffbd5f2cab6e5dbe9822cd1e\": container with ID starting with d63cecc41ad759491c5e5533bb495d3f39518fc6ffbd5f2cab6e5dbe9822cd1e not found: ID does not exist"
Nov 02 20:30:11.318394 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.318351    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-data\") pod \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") "
Nov 02 20:30:11.318611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.318408    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-credential-socket\") pod \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") "
Nov 02 20:30:11.318611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.318473    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-podinfo\") pod \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") "
Nov 02 20:30:11.318611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.318540    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-kube-api-access-wfmgj\" (UniqueName: \"kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-kube-api-access-wfmgj\") pod \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") "
Nov 02 20:30:11.318611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.318565    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-token\" (UniqueName: \"kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-token\") pod \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") "
Nov 02 20:30:11.318873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.318706    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-socket\") pod \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") "
Nov 02 20:30:11.318873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.318718    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-credential-socket" pod "8c802b4b-1dec-400b-a1e7-5507933ef8e3" (UID: "8c802b4b-1dec-400b-a1e7-5507933ef8e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:30:11.318873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.318731    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-data" pod "8c802b4b-1dec-400b-a1e7-5507933ef8e3" (UID: "8c802b4b-1dec-400b-a1e7-5507933ef8e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:30:11.318873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.318800    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istiod-ca-cert\") pod \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") "
Nov 02 20:30:11.318873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.318864    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-envoy\") pod \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") "
Nov 02 20:30:11.319137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.318968    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-certs\") pod \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\" (UID: \"8c802b4b-1dec-400b-a1e7-5507933ef8e3\") "
Nov 02 20:30:11.319137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.319059    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-socket" pod "8c802b4b-1dec-400b-a1e7-5507933ef8e3" (UID: "8c802b4b-1dec-400b-a1e7-5507933ef8e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:30:11.319258 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.319154    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istiod-ca-cert" pod "8c802b4b-1dec-400b-a1e7-5507933ef8e3" (UID: "8c802b4b-1dec-400b-a1e7-5507933ef8e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:11.319311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.319263    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-certs" pod "8c802b4b-1dec-400b-a1e7-5507933ef8e3" (UID: "8c802b4b-1dec-400b-a1e7-5507933ef8e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:30:11.319508 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.319488    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:11.319571 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.319512    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:11.319571 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.319521    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:11.319571 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.319530    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:11.319571 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.319544    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:11.321463 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.321432    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-podinfo" pod "8c802b4b-1dec-400b-a1e7-5507933ef8e3" (UID: "8c802b4b-1dec-400b-a1e7-5507933ef8e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:30:11.321463 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.321461    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-kube-api-access-wfmgj" pod "8c802b4b-1dec-400b-a1e7-5507933ef8e3" (UID: "8c802b4b-1dec-400b-a1e7-5507933ef8e3"). InnerVolumeSpecName "kube-api-access-wfmgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:11.321640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.321524    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-envoy" pod "8c802b4b-1dec-400b-a1e7-5507933ef8e3" (UID: "8c802b4b-1dec-400b-a1e7-5507933ef8e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:30:11.321640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.321533    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-token" pod "8c802b4b-1dec-400b-a1e7-5507933ef8e3" (UID: "8c802b4b-1dec-400b-a1e7-5507933ef8e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:11.420509 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.420415    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:11.420509 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.420450    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:11.420509 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.420461    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfmgj\" (UniqueName: \"kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-kube-api-access-wfmgj\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:11.420509 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.420472    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c802b4b-1dec-400b-a1e7-5507933ef8e3-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:11.672983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.672889    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"]
Nov 02 20:30:11.675968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:11.675932    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4ms8z"]
Nov 02 20:30:13.160735 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:13.160699    5224 scope.go:122] "RemoveContainer" containerID="3c82ec334f30bb724509c1423714a63fb014d7656864a62ea17ec7e9f8910a26"
Nov 02 20:30:13.167930 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:13.167900    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" path="/var/lib/kubelet/pods/8c802b4b-1dec-400b-a1e7-5507933ef8e3/volumes"
Nov 02 20:30:13.915744 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:13.915706    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:14.161523 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:14.161485    5224 scope.go:122] "RemoveContainer" containerID="78f09992e913c364c60bf14493c783e10bac2a026cbcf7ddb40893d3d03ae31d"
Nov 02 20:30:14.268093 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:14.268059    5224 generic.go:356] "Generic (PLEG): container finished" podID="201c4e04-0153-4013-978c-2d8be535a640" containerID="78119e1f3d8cbfd05d127ff1500f4f516da2b41218d1ffee0b4c7c327a420236" exitCode=60
Nov 02 20:30:14.268257 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:14.268131    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8" event={"ID":"201c4e04-0153-4013-978c-2d8be535a640","Type":"ContainerDied","Data":"78119e1f3d8cbfd05d127ff1500f4f516da2b41218d1ffee0b4c7c327a420236"}
Nov 02 20:30:14.268257 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:14.268195    5224 scope.go:122] "RemoveContainer" containerID="3c82ec334f30bb724509c1423714a63fb014d7656864a62ea17ec7e9f8910a26"
Nov 02 20:30:14.268594 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:14.268576    5224 scope.go:122] "RemoveContainer" containerID="78119e1f3d8cbfd05d127ff1500f4f516da2b41218d1ffee0b4c7c327a420236"
Nov 02 20:30:14.268960 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:14.268928    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8_odh-ai-gateway-infra(201c4e04-0153-4013-978c-2d8be535a640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8" podUID="201c4e04-0153-4013-978c-2d8be535a640"
Nov 02 20:30:14.981911 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:14.981874    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:15.161468 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:15.161434    5224 scope.go:122] "RemoveContainer" containerID="0143c8462cde35d63d9033fde1cec668aeab02468ef6a119f1ee554c6cf44269"
Nov 02 20:30:15.277829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:15.277792    5224 generic.go:356] "Generic (PLEG): container finished" podID="1c75ab44-108b-4a6c-8a99-8ee871f8894a" containerID="25c329a7acef147185dd6ffabe3c31ea9ac3ed63142c778304ff59422b503bb4" exitCode=60
Nov 02 20:30:15.278364 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:15.277882    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h" event={"ID":"1c75ab44-108b-4a6c-8a99-8ee871f8894a","Type":"ContainerDied","Data":"25c329a7acef147185dd6ffabe3c31ea9ac3ed63142c778304ff59422b503bb4"}
Nov 02 20:30:15.278364 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:15.277952    5224 scope.go:122] "RemoveContainer" containerID="78f09992e913c364c60bf14493c783e10bac2a026cbcf7ddb40893d3d03ae31d"
Nov 02 20:30:15.278458 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:15.278410    5224 scope.go:122] "RemoveContainer" containerID="25c329a7acef147185dd6ffabe3c31ea9ac3ed63142c778304ff59422b503bb4"
Nov 02 20:30:15.278827 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:15.278752    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h_odh-ai-gateway-infra(1c75ab44-108b-4a6c-8a99-8ee871f8894a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h" podUID="1c75ab44-108b-4a6c-8a99-8ee871f8894a"
Nov 02 20:30:15.393548 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:15.393506    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"]
Nov 02 20:30:15.570869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:15.570834    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"
Nov 02 20:30:15.667686 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:15.667650    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/201c4e04-0153-4013-978c-2d8be535a640-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/201c4e04-0153-4013-978c-2d8be535a640-maas-api-service-ca\") pod \"201c4e04-0153-4013-978c-2d8be535a640\" (UID: \"201c4e04-0153-4013-978c-2d8be535a640\") "
Nov 02 20:30:15.667901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:15.667708    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201c4e04-0153-4013-978c-2d8be535a640-kube-api-access-d45mv\" (UniqueName: \"kubernetes.io/projected/201c4e04-0153-4013-978c-2d8be535a640-kube-api-access-d45mv\") pod \"201c4e04-0153-4013-978c-2d8be535a640\" (UID: \"201c4e04-0153-4013-978c-2d8be535a640\") "
Nov 02 20:30:15.668069 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:15.668042    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201c4e04-0153-4013-978c-2d8be535a640-maas-api-service-ca" pod "201c4e04-0153-4013-978c-2d8be535a640" (UID: "201c4e04-0153-4013-978c-2d8be535a640"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:15.668281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:15.668264    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/201c4e04-0153-4013-978c-2d8be535a640-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:15.669861 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:15.669836    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201c4e04-0153-4013-978c-2d8be535a640-kube-api-access-d45mv" pod "201c4e04-0153-4013-978c-2d8be535a640" (UID: "201c4e04-0153-4013-978c-2d8be535a640"). InnerVolumeSpecName "kube-api-access-d45mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:15.769110 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:15.769064    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d45mv\" (UniqueName: \"kubernetes.io/projected/201c4e04-0153-4013-978c-2d8be535a640-kube-api-access-d45mv\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:16.288102 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.288064    5224 generic.go:356] "Generic (PLEG): container finished" podID="0c254164-9b88-49e9-9e90-575daf8b855e" containerID="e0a38e73156d4f5f31bc3d9a725b1b4207b445fe10d8042c6703fd5385b97186" exitCode=60
Nov 02 20:30:16.288610 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.288143    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr" event={"ID":"0c254164-9b88-49e9-9e90-575daf8b855e","Type":"ContainerDied","Data":"e0a38e73156d4f5f31bc3d9a725b1b4207b445fe10d8042c6703fd5385b97186"}
Nov 02 20:30:16.288610 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.288228    5224 scope.go:122] "RemoveContainer" containerID="0143c8462cde35d63d9033fde1cec668aeab02468ef6a119f1ee554c6cf44269"
Nov 02 20:30:16.288733 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.288657    5224 scope.go:122] "RemoveContainer" containerID="e0a38e73156d4f5f31bc3d9a725b1b4207b445fe10d8042c6703fd5385b97186"
Nov 02 20:30:16.289040 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:16.289014    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr_odh-ai-gateway-infra(0c254164-9b88-49e9-9e90-575daf8b855e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr" podUID="0c254164-9b88-49e9-9e90-575daf8b855e"
Nov 02 20:30:16.290162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.290140    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"
Nov 02 20:30:16.290283 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.290190    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8" event={"ID":"201c4e04-0153-4013-978c-2d8be535a640","Type":"ContainerDied","Data":"efcbde59ca211dfde9b40490c586ae0e9f73443f7c785dd6346dfa32b64895f5"}
Nov 02 20:30:16.301697 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.301670    5224 scope.go:122] "RemoveContainer" containerID="78119e1f3d8cbfd05d127ff1500f4f516da2b41218d1ffee0b4c7c327a420236"
Nov 02 20:30:16.406015 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.405978    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"]
Nov 02 20:30:16.532790 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.522353    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"]
Nov 02 20:30:16.532790 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.526599    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894190-sb8w8"]
Nov 02 20:30:16.570997 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.570921    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"
Nov 02 20:30:16.645387 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.645333    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:16.680511 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.680471    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c75ab44-108b-4a6c-8a99-8ee871f8894a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c75ab44-108b-4a6c-8a99-8ee871f8894a-maas-api-service-ca\") pod \"1c75ab44-108b-4a6c-8a99-8ee871f8894a\" (UID: \"1c75ab44-108b-4a6c-8a99-8ee871f8894a\") "
Nov 02 20:30:16.680711 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.680557    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c75ab44-108b-4a6c-8a99-8ee871f8894a-kube-api-access-znt5h\" (UniqueName: \"kubernetes.io/projected/1c75ab44-108b-4a6c-8a99-8ee871f8894a-kube-api-access-znt5h\") pod \"1c75ab44-108b-4a6c-8a99-8ee871f8894a\" (UID: \"1c75ab44-108b-4a6c-8a99-8ee871f8894a\") "
Nov 02 20:30:16.680974 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.680947    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c75ab44-108b-4a6c-8a99-8ee871f8894a-maas-api-service-ca" pod "1c75ab44-108b-4a6c-8a99-8ee871f8894a" (UID: "1c75ab44-108b-4a6c-8a99-8ee871f8894a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:16.682806 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.682782    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c75ab44-108b-4a6c-8a99-8ee871f8894a-kube-api-access-znt5h" pod "1c75ab44-108b-4a6c-8a99-8ee871f8894a" (UID: "1c75ab44-108b-4a6c-8a99-8ee871f8894a"). InnerVolumeSpecName "kube-api-access-znt5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:16.782559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.782514    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c75ab44-108b-4a6c-8a99-8ee871f8894a-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:16.782559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.782552    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znt5h\" (UniqueName: \"kubernetes.io/projected/1c75ab44-108b-4a6c-8a99-8ee871f8894a-kube-api-access-znt5h\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:16.822898 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:16.822761    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:17.160706 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.160597    5224 scope.go:122] "RemoveContainer" containerID="d3de42d6d667f4924dc2c7a569abe11ec4cc3354e1ebdf6b81cfad03b9fa3bbf"
Nov 02 20:30:17.160920 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.160871    5224 scope.go:122] "RemoveContainer" containerID="45f669164d9affe785ab7fa95cf283864c290eaaea2aca938fc79e1502f53909"
Nov 02 20:30:17.168811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.168758    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201c4e04-0153-4013-978c-2d8be535a640" path="/var/lib/kubelet/pods/201c4e04-0153-4013-978c-2d8be535a640/volumes"
Nov 02 20:30:17.301690 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.301653    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"
Nov 02 20:30:17.302221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.301659    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h" event={"ID":"1c75ab44-108b-4a6c-8a99-8ee871f8894a","Type":"ContainerDied","Data":"0e8b0d2ffb959c8d675a8c041a40f7c193ea124714c6c84ea9bbcd78936bb406"}
Nov 02 20:30:17.302221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.301823    5224 scope.go:122] "RemoveContainer" containerID="25c329a7acef147185dd6ffabe3c31ea9ac3ed63142c778304ff59422b503bb4"
Nov 02 20:30:17.418041 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.417952    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"]
Nov 02 20:30:17.470341 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.470309    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"]
Nov 02 20:30:17.472524 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.472488    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894190-z6m2h"]
Nov 02 20:30:17.573689 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.573661    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"
Nov 02 20:30:17.592665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.592611    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c254164-9b88-49e9-9e90-575daf8b855e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c254164-9b88-49e9-9e90-575daf8b855e-maas-api-service-ca\") pod \"0c254164-9b88-49e9-9e90-575daf8b855e\" (UID: \"0c254164-9b88-49e9-9e90-575daf8b855e\") "
Nov 02 20:30:17.592912 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.592806    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c254164-9b88-49e9-9e90-575daf8b855e-kube-api-access-5xlrb\" (UniqueName: \"kubernetes.io/projected/0c254164-9b88-49e9-9e90-575daf8b855e-kube-api-access-5xlrb\") pod \"0c254164-9b88-49e9-9e90-575daf8b855e\" (UID: \"0c254164-9b88-49e9-9e90-575daf8b855e\") "
Nov 02 20:30:17.593095 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.593069    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c254164-9b88-49e9-9e90-575daf8b855e-maas-api-service-ca" pod "0c254164-9b88-49e9-9e90-575daf8b855e" (UID: "0c254164-9b88-49e9-9e90-575daf8b855e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:17.593238 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.593222    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c254164-9b88-49e9-9e90-575daf8b855e-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:17.595018 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.594991    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c254164-9b88-49e9-9e90-575daf8b855e-kube-api-access-5xlrb" pod "0c254164-9b88-49e9-9e90-575daf8b855e" (UID: "0c254164-9b88-49e9-9e90-575daf8b855e"). InnerVolumeSpecName "kube-api-access-5xlrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:17.694868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:17.694741    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xlrb\" (UniqueName: \"kubernetes.io/projected/0c254164-9b88-49e9-9e90-575daf8b855e-kube-api-access-5xlrb\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:18.183393 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:18.183354    5224 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3934517c_ae41_4e8e_9651_dda812f7daea.slice/crio-conmon-aa12420da8e7827a58d1ff5cb80e27067ee2e203823deecee887ddd4e69c50db.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 20:30:18.222018 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.221944    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:18.312212 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.312116    5224 generic.go:356] "Generic (PLEG): container finished" podID="068b0dec-2759-4b13-ab24-090ee50010de" containerID="0cd133421852ab1cbc014bfccfbe112a426dc7b9011712b2089cf08234a117a9" exitCode=60
Nov 02 20:30:18.312212 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.312188    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm" event={"ID":"068b0dec-2759-4b13-ab24-090ee50010de","Type":"ContainerDied","Data":"0cd133421852ab1cbc014bfccfbe112a426dc7b9011712b2089cf08234a117a9"}
Nov 02 20:30:18.312743 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.312260    5224 scope.go:122] "RemoveContainer" containerID="d3de42d6d667f4924dc2c7a569abe11ec4cc3354e1ebdf6b81cfad03b9fa3bbf"
Nov 02 20:30:18.312743 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.312690    5224 scope.go:122] "RemoveContainer" containerID="0cd133421852ab1cbc014bfccfbe112a426dc7b9011712b2089cf08234a117a9"
Nov 02 20:30:18.313113 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:18.313088    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm_odh-ai-gateway-infra(068b0dec-2759-4b13-ab24-090ee50010de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm" podUID="068b0dec-2759-4b13-ab24-090ee50010de"
Nov 02 20:30:18.314733 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.314709    5224 generic.go:356] "Generic (PLEG): container finished" podID="3934517c-ae41-4e8e-9651-dda812f7daea" containerID="aa12420da8e7827a58d1ff5cb80e27067ee2e203823deecee887ddd4e69c50db" exitCode=60
Nov 02 20:30:18.314889 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.314798    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr" event={"ID":"3934517c-ae41-4e8e-9651-dda812f7daea","Type":"ContainerDied","Data":"aa12420da8e7827a58d1ff5cb80e27067ee2e203823deecee887ddd4e69c50db"}
Nov 02 20:30:18.315258 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.315237    5224 scope.go:122] "RemoveContainer" containerID="aa12420da8e7827a58d1ff5cb80e27067ee2e203823deecee887ddd4e69c50db"
Nov 02 20:30:18.315508 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:18.315487    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr_odh-ai-gateway-infra(3934517c-ae41-4e8e-9651-dda812f7daea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr" podUID="3934517c-ae41-4e8e-9651-dda812f7daea"
Nov 02 20:30:18.317923 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.317902    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr" event={"ID":"0c254164-9b88-49e9-9e90-575daf8b855e","Type":"ContainerDied","Data":"c35072ce429458b6fc72a70b749c8f683d8beb3832de1b716392373a41b2b585"}
Nov 02 20:30:18.318029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.317937    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"
Nov 02 20:30:18.327254 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.327215    5224 scope.go:122] "RemoveContainer" containerID="45f669164d9affe785ab7fa95cf283864c290eaaea2aca938fc79e1502f53909"
Nov 02 20:30:18.340657 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.340634    5224 scope.go:122] "RemoveContainer" containerID="e0a38e73156d4f5f31bc3d9a725b1b4207b445fe10d8042c6703fd5385b97186"
Nov 02 20:30:18.660423 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.660208    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"]
Nov 02 20:30:18.665095 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.665052    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-93816f-29894190-6hwcr"]
Nov 02 20:30:18.758438 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:18.758384    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:19.167148 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.167107    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c254164-9b88-49e9-9e90-575daf8b855e" path="/var/lib/kubelet/pods/0c254164-9b88-49e9-9e90-575daf8b855e/volumes"
Nov 02 20:30:19.167554 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.167540    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c75ab44-108b-4a6c-8a99-8ee871f8894a" path="/var/lib/kubelet/pods/1c75ab44-108b-4a6c-8a99-8ee871f8894a/volumes"
Nov 02 20:30:19.439681 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.439579    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"]
Nov 02 20:30:19.552084 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.552044    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"]
Nov 02 20:30:19.594521 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.594491    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"
Nov 02 20:30:19.716983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.716955    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"
Nov 02 20:30:19.718994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.718972    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/068b0dec-2759-4b13-ab24-090ee50010de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068b0dec-2759-4b13-ab24-090ee50010de-maas-api-service-ca\") pod \"068b0dec-2759-4b13-ab24-090ee50010de\" (UID: \"068b0dec-2759-4b13-ab24-090ee50010de\") "
Nov 02 20:30:19.719067 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.719042    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068b0dec-2759-4b13-ab24-090ee50010de-kube-api-access-7lb8z\" (UniqueName: \"kubernetes.io/projected/068b0dec-2759-4b13-ab24-090ee50010de-kube-api-access-7lb8z\") pod \"068b0dec-2759-4b13-ab24-090ee50010de\" (UID: \"068b0dec-2759-4b13-ab24-090ee50010de\") "
Nov 02 20:30:19.719416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.719386    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068b0dec-2759-4b13-ab24-090ee50010de-maas-api-service-ca" pod "068b0dec-2759-4b13-ab24-090ee50010de" (UID: "068b0dec-2759-4b13-ab24-090ee50010de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:19.719586 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.719571    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068b0dec-2759-4b13-ab24-090ee50010de-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:19.721221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.721197    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068b0dec-2759-4b13-ab24-090ee50010de-kube-api-access-7lb8z" pod "068b0dec-2759-4b13-ab24-090ee50010de" (UID: "068b0dec-2759-4b13-ab24-090ee50010de"). InnerVolumeSpecName "kube-api-access-7lb8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:19.820789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.820733    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3934517c-ae41-4e8e-9651-dda812f7daea-kube-api-access-4ww9c\" (UniqueName: \"kubernetes.io/projected/3934517c-ae41-4e8e-9651-dda812f7daea-kube-api-access-4ww9c\") pod \"3934517c-ae41-4e8e-9651-dda812f7daea\" (UID: \"3934517c-ae41-4e8e-9651-dda812f7daea\") "
Nov 02 20:30:19.820985 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.820826    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3934517c-ae41-4e8e-9651-dda812f7daea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3934517c-ae41-4e8e-9651-dda812f7daea-maas-api-service-ca\") pod \"3934517c-ae41-4e8e-9651-dda812f7daea\" (UID: \"3934517c-ae41-4e8e-9651-dda812f7daea\") "
Nov 02 20:30:19.821132 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.821116    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lb8z\" (UniqueName: \"kubernetes.io/projected/068b0dec-2759-4b13-ab24-090ee50010de-kube-api-access-7lb8z\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:19.821257 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.821233    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3934517c-ae41-4e8e-9651-dda812f7daea-maas-api-service-ca" pod "3934517c-ae41-4e8e-9651-dda812f7daea" (UID: "3934517c-ae41-4e8e-9651-dda812f7daea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:19.823355 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.823324    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3934517c-ae41-4e8e-9651-dda812f7daea-kube-api-access-4ww9c" pod "3934517c-ae41-4e8e-9651-dda812f7daea" (UID: "3934517c-ae41-4e8e-9651-dda812f7daea"). InnerVolumeSpecName "kube-api-access-4ww9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:19.922259 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.922220    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ww9c\" (UniqueName: \"kubernetes.io/projected/3934517c-ae41-4e8e-9651-dda812f7daea-kube-api-access-4ww9c\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:19.922259 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:19.922253    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3934517c-ae41-4e8e-9651-dda812f7daea-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:20.336694 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:20.336660    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"
Nov 02 20:30:20.336947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:20.336654    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm" event={"ID":"068b0dec-2759-4b13-ab24-090ee50010de","Type":"ContainerDied","Data":"2e9bd6e36f2dccc9e532ad9eaed3bd66c057b9a3653f0dc0f207c6e24c39c6ba"}
Nov 02 20:30:20.336947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:20.336854    5224 scope.go:122] "RemoveContainer" containerID="0cd133421852ab1cbc014bfccfbe112a426dc7b9011712b2089cf08234a117a9"
Nov 02 20:30:20.338306 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:20.338275    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr" event={"ID":"3934517c-ae41-4e8e-9651-dda812f7daea","Type":"ContainerDied","Data":"27ba4a7b807194ec35eb5d40ed9d5c01dacee39d96851e5c52b58b56766d71b6"}
Nov 02 20:30:20.338471 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:20.338323    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"
Nov 02 20:30:20.348613 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:20.348191    5224 scope.go:122] "RemoveContainer" containerID="aa12420da8e7827a58d1ff5cb80e27067ee2e203823deecee887ddd4e69c50db"
Nov 02 20:30:20.505457 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:20.505417    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"]
Nov 02 20:30:20.508644 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:20.508612    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-408103-29894190-n54qr"]
Nov 02 20:30:20.618795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:20.618689    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"]
Nov 02 20:30:20.622241 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:20.622216    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894190-jr6pm"]
Nov 02 20:30:21.167882 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:21.167842    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068b0dec-2759-4b13-ab24-090ee50010de" path="/var/lib/kubelet/pods/068b0dec-2759-4b13-ab24-090ee50010de/volumes"
Nov 02 20:30:21.168292 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:21.168279    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3934517c-ae41-4e8e-9651-dda812f7daea" path="/var/lib/kubelet/pods/3934517c-ae41-4e8e-9651-dda812f7daea/volumes"
Nov 02 20:30:21.175137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:21.175099    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:21.453470 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:21.453374    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:22.268384 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:22.268299    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:22.745303 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:22.745199    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:23.282510 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:23.282451    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:24.058369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:24.058330    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:26.731104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:26.731064    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"]
Nov 02 20:30:26.731609 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:26.731302    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg" podUID="b2003aed-acbb-4307-9733-fd6e0b791217" containerName="maas-api" containerID="cri-o://e8d7b222b150a62debccd7cc341d16d161a4c4a363fd07e91832e4b95b0b98b8" gracePeriod=30
Nov 02 20:30:26.874015 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:26.873969    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:27.034075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.034045    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:30:27.100922 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.100872    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2003aed-acbb-4307-9733-fd6e0b791217-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2003aed-acbb-4307-9733-fd6e0b791217-openshift-service-ca\") pod \"b2003aed-acbb-4307-9733-fd6e0b791217\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") "
Nov 02 20:30:27.101122 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.101039    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-metrics-certs\") pod \"b2003aed-acbb-4307-9733-fd6e0b791217\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") "
Nov 02 20:30:27.101122 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.101076    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2003aed-acbb-4307-9733-fd6e0b791217-kube-api-access-vsqlx\" (UniqueName: \"kubernetes.io/projected/b2003aed-acbb-4307-9733-fd6e0b791217-kube-api-access-vsqlx\") pod \"b2003aed-acbb-4307-9733-fd6e0b791217\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") "
Nov 02 20:30:27.101122 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.101107    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-maas-api-tls\") pod \"b2003aed-acbb-4307-9733-fd6e0b791217\" (UID: \"b2003aed-acbb-4307-9733-fd6e0b791217\") "
Nov 02 20:30:27.101333 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.101295    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2003aed-acbb-4307-9733-fd6e0b791217-openshift-service-ca" pod "b2003aed-acbb-4307-9733-fd6e0b791217" (UID: "b2003aed-acbb-4307-9733-fd6e0b791217"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:27.101623 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.101595    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2003aed-acbb-4307-9733-fd6e0b791217-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:27.103914 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.103878    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-maas-api-tls" pod "b2003aed-acbb-4307-9733-fd6e0b791217" (UID: "b2003aed-acbb-4307-9733-fd6e0b791217"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:30:27.103914 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.103902    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2003aed-acbb-4307-9733-fd6e0b791217-kube-api-access-vsqlx" pod "b2003aed-acbb-4307-9733-fd6e0b791217" (UID: "b2003aed-acbb-4307-9733-fd6e0b791217"). InnerVolumeSpecName "kube-api-access-vsqlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:27.104150 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.103946    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-metrics-certs" pod "b2003aed-acbb-4307-9733-fd6e0b791217" (UID: "b2003aed-acbb-4307-9733-fd6e0b791217"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:30:27.202874 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.202834    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:27.202874 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.202865    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsqlx\" (UniqueName: \"kubernetes.io/projected/b2003aed-acbb-4307-9733-fd6e0b791217-kube-api-access-vsqlx\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:27.202874 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.202886    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2003aed-acbb-4307-9733-fd6e0b791217-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:27.369652 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:27.369608    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:30:27.386922 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.386885    5224 generic.go:356] "Generic (PLEG): container finished" podID="b2003aed-acbb-4307-9733-fd6e0b791217" containerID="e8d7b222b150a62debccd7cc341d16d161a4c4a363fd07e91832e4b95b0b98b8" exitCode=0
Nov 02 20:30:27.387109 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.386941    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"
Nov 02 20:30:27.387109 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.386971    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg" event={"ID":"b2003aed-acbb-4307-9733-fd6e0b791217","Type":"ContainerDied","Data":"e8d7b222b150a62debccd7cc341d16d161a4c4a363fd07e91832e4b95b0b98b8"}
Nov 02 20:30:27.387109 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.387039    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg" event={"ID":"b2003aed-acbb-4307-9733-fd6e0b791217","Type":"ContainerDied","Data":"38e6ca7899ae4879568a534f4f7e5b1db1b19b46e0167ecbeca54d2b898597ed"}
Nov 02 20:30:27.387109 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.387073    5224 scope.go:122] "RemoveContainer" containerID="e8d7b222b150a62debccd7cc341d16d161a4c4a363fd07e91832e4b95b0b98b8"
Nov 02 20:30:27.396758 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.396730    5224 scope.go:122] "RemoveContainer" containerID="e8d7b222b150a62debccd7cc341d16d161a4c4a363fd07e91832e4b95b0b98b8"
Nov 02 20:30:27.397112 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:27.397087    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d7b222b150a62debccd7cc341d16d161a4c4a363fd07e91832e4b95b0b98b8\": container with ID starting with e8d7b222b150a62debccd7cc341d16d161a4c4a363fd07e91832e4b95b0b98b8 not found: ID does not exist" containerID="e8d7b222b150a62debccd7cc341d16d161a4c4a363fd07e91832e4b95b0b98b8"
Nov 02 20:30:27.397165 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.397128    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d7b222b150a62debccd7cc341d16d161a4c4a363fd07e91832e4b95b0b98b8"} err="failed to get container status \"e8d7b222b150a62debccd7cc341d16d161a4c4a363fd07e91832e4b95b0b98b8\": rpc error: code = NotFound desc = could not find container \"e8d7b222b150a62debccd7cc341d16d161a4c4a363fd07e91832e4b95b0b98b8\": container with ID starting with e8d7b222b150a62debccd7cc341d16d161a4c4a363fd07e91832e4b95b0b98b8 not found: ID does not exist"
Nov 02 20:30:27.444970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.444927    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:27.557410 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.557374    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"]
Nov 02 20:30:27.560782 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.560741    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-408103-565c587697-rcntg"]
Nov 02 20:30:27.616416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:27.616372    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:28.540901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:28.540860    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:28.765533 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:28.765488    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:29.167708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:29.167666    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2003aed-acbb-4307-9733-fd6e0b791217" path="/var/lib/kubelet/pods/b2003aed-acbb-4307-9733-fd6e0b791217/volumes"
Nov 02 20:30:29.845585 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:29.845538    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:30.023059 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:30.023021    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:30.210242 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:30.210138    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:30.384861 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:30.384794    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:30.679835 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:30.679781    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:32.074375 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:32.074334    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:32.624750 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:32.624702    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:32.924511 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:32.924401    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:34.301093 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:34.301056    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:35.869400 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:35.869354    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:36.628085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:36.628045    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:36.841893 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:36.841855    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:37.015534 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:37.015499    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:38.957203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:38.957168    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:30:39.699215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:39.699163    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:39.899821 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:39.893556    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:40.091140 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:40.091095    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:45.122521 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:45.122472    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:46.719163 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:46.719121    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:46.893624 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:46.893591    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:50.275489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:50.275359    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:53.729333 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:53.729288    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:54.862480 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:54.862440    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:55.467397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:55.467351    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:55.715938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:55.715888    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:55.896538 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:55.896499    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:57.360287 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:30:57.360242    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:30:58.952116 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:58.952079    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:59.709236 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:59.709195    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:30:59.935412 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:30:59.935371    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:00.112553 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:00.112512    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:00.609100 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:00.609061    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:02.997210 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:02.997161    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:03.173629 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:03.173584    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:06.013234 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:06.013161    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:07.635793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:07.635721    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:31:11.197908 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:11.197861    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:12.921077 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:12.921034    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:15.929635 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:15.929597    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:16.104132 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:16.104079    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:17.892178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:17.892137    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:21.612437 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:21.612394    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:22.604307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:22.604263    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:22.824188 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:22.824148    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:23.305696 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:23.305638    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:24.686608 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:24.686568    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:24.862649 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:24.862605    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:26.235536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:26.235489    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:26.771539 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:26.771495    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:27.362492 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:31:27.362449    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:31:27.392623 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:27.392587    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:28.495089 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:28.495046    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:28.672565 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:28.672516    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:28.839093 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:28.839051    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:33.895064 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:33.895026    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:38.965232 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:38.965184    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:43.040380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.040338    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"]
Nov 02 20:31:43.040916 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.040558    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb" podUID="0543f1ee-7bcb-491e-94b0-996ec5133c2f" containerName="maas-api" containerID="cri-o://ea61d1281fde7ede3307f357275aceecc0e7c2717aaf35e04d43782cac34164a" gracePeriod=30
Nov 02 20:31:43.179762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.179719    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:43.333254 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.333215    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:31:43.471471 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.471437    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0543f1ee-7bcb-491e-94b0-996ec5133c2f-kube-api-access-56rwv\" (UniqueName: \"kubernetes.io/projected/0543f1ee-7bcb-491e-94b0-996ec5133c2f-kube-api-access-56rwv\") pod \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") "
Nov 02 20:31:43.471683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.471505    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-metrics-certs\") pod \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") "
Nov 02 20:31:43.471683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.471548    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-maas-api-tls\") pod \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") "
Nov 02 20:31:43.471683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.471604    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0543f1ee-7bcb-491e-94b0-996ec5133c2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0543f1ee-7bcb-491e-94b0-996ec5133c2f-openshift-service-ca\") pod \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\" (UID: \"0543f1ee-7bcb-491e-94b0-996ec5133c2f\") "
Nov 02 20:31:43.472310 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.472269    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0543f1ee-7bcb-491e-94b0-996ec5133c2f-openshift-service-ca" pod "0543f1ee-7bcb-491e-94b0-996ec5133c2f" (UID: "0543f1ee-7bcb-491e-94b0-996ec5133c2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:31:43.474248 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.474209    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0543f1ee-7bcb-491e-94b0-996ec5133c2f-kube-api-access-56rwv" pod "0543f1ee-7bcb-491e-94b0-996ec5133c2f" (UID: "0543f1ee-7bcb-491e-94b0-996ec5133c2f"). InnerVolumeSpecName "kube-api-access-56rwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:43.474396 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.474375    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-maas-api-tls" pod "0543f1ee-7bcb-491e-94b0-996ec5133c2f" (UID: "0543f1ee-7bcb-491e-94b0-996ec5133c2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:31:43.475429 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.475395    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-metrics-certs" pod "0543f1ee-7bcb-491e-94b0-996ec5133c2f" (UID: "0543f1ee-7bcb-491e-94b0-996ec5133c2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:31:43.573567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.573479    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0543f1ee-7bcb-491e-94b0-996ec5133c2f-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:43.573567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.573511    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56rwv\" (UniqueName: \"kubernetes.io/projected/0543f1ee-7bcb-491e-94b0-996ec5133c2f-kube-api-access-56rwv\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:43.573567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.573523    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:43.573567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.573531    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0543f1ee-7bcb-491e-94b0-996ec5133c2f-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:43.838828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.838726    5224 generic.go:356] "Generic (PLEG): container finished" podID="0543f1ee-7bcb-491e-94b0-996ec5133c2f" containerID="ea61d1281fde7ede3307f357275aceecc0e7c2717aaf35e04d43782cac34164a" exitCode=0
Nov 02 20:31:43.838828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.838786    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"
Nov 02 20:31:43.838828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.838816    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb" event={"ID":"0543f1ee-7bcb-491e-94b0-996ec5133c2f","Type":"ContainerDied","Data":"ea61d1281fde7ede3307f357275aceecc0e7c2717aaf35e04d43782cac34164a"}
Nov 02 20:31:43.839085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.838875    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb" event={"ID":"0543f1ee-7bcb-491e-94b0-996ec5133c2f","Type":"ContainerDied","Data":"c28baf91a065b29cf5dbd6a2f6060e0966c4aa367a1d421000c2c4c2b70bcc53"}
Nov 02 20:31:43.839085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.838902    5224 scope.go:122] "RemoveContainer" containerID="ea61d1281fde7ede3307f357275aceecc0e7c2717aaf35e04d43782cac34164a"
Nov 02 20:31:43.849482 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.849460    5224 scope.go:122] "RemoveContainer" containerID="ea61d1281fde7ede3307f357275aceecc0e7c2717aaf35e04d43782cac34164a"
Nov 02 20:31:43.849821 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:31:43.849801    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea61d1281fde7ede3307f357275aceecc0e7c2717aaf35e04d43782cac34164a\": container with ID starting with ea61d1281fde7ede3307f357275aceecc0e7c2717aaf35e04d43782cac34164a not found: ID does not exist" containerID="ea61d1281fde7ede3307f357275aceecc0e7c2717aaf35e04d43782cac34164a"
Nov 02 20:31:43.849864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:43.849832    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea61d1281fde7ede3307f357275aceecc0e7c2717aaf35e04d43782cac34164a"} err="failed to get container status \"ea61d1281fde7ede3307f357275aceecc0e7c2717aaf35e04d43782cac34164a\": rpc error: code = NotFound desc = could not find container \"ea61d1281fde7ede3307f357275aceecc0e7c2717aaf35e04d43782cac34164a\": container with ID starting with ea61d1281fde7ede3307f357275aceecc0e7c2717aaf35e04d43782cac34164a not found: ID does not exist"
Nov 02 20:31:44.006951 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:44.006908    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"]
Nov 02 20:31:44.012016 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:44.011968    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7cad76-bdc56f996-2dllb"]
Nov 02 20:31:44.964471 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:44.964425    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:45.167626 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:45.167589    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0543f1ee-7bcb-491e-94b0-996ec5133c2f" path="/var/lib/kubelet/pods/0543f1ee-7bcb-491e-94b0-996ec5133c2f/volumes"
Nov 02 20:31:49.743365 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:49.743325    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:49.964721 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:49.964668    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:50.140395 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:50.140357    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:50.372331 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:50.372287    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:50.552316 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:50.552275    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:51.866391 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:51.866347    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:52.413919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:52.413875    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:53.201956 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:53.201920    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:54.281318 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:54.281280    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:31:55.410053 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:55.410008    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:57.360606 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:31:57.360568    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:31:58.691711 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:58.691676    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:58.909837 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:58.909805    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:59.271696 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:59.271657    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:31:59.443968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:31:59.443928    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:00.192261 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:00.192225    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:02.247932 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:02.247884    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:04.924550 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:04.924500    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:07.574174 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:07.574126    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:08.253071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:08.253026    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:08.418044 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:08.417995    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:12.611065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:12.611003    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:17.579632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:17.579585    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:22.594648 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:22.594541    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:23.460361 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:23.460318    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:24.550504 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:24.550456    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:25.166994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:25.166947    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:25.388757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:25.388701    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:27.360880 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:32:27.360844    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:32:27.878256 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:27.878192    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:29.401309 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:29.401269    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:29.760022 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:29.759927    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:32.781115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:32.781076    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:32.958111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:32.958074    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:33.819442 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:33.819401    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:35.649911 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:35.649873    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:32:38.734541 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:38.734478    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:43.652437 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:43.652391    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:47.014071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:47.013952    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:50.219915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:50.219876    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:50.786112 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:50.786076    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:51.289902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:51.289861    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:53.804005 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:53.803962    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:53.976150 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:53.976108    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:55.097542 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:55.097463    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:55.392178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:55.392083    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:55.907180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:55.907142    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:57.027823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:57.027747    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:57.358970 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:32:57.358882    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:32:58.305106 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:58.305045    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:58.522010 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:58.521968    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:32:59.593226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:32:59.593186    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:01.786567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:01.786527    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:02.586725 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:02.586670    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:05.842130 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:05.842086    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:10.752748 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:10.752708    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:15.506928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:15.506885    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:15.729709 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:15.729670    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:15.919964 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:15.919924    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:19.954169 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:19.954128    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"]
Nov 02 20:33:19.954840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:19.954553    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g" podUID="851939a7-2f00-4158-9cfb-9a43641a5c5e" containerName="maas-api" containerID="cri-o://2b38805d0487dadef5e381a108480079cceacc30c71ca99e3cfb196a09e67c0e" gracePeriod=30
Nov 02 20:33:20.094055 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.094015    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:20.245185 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.245146    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:33:20.370477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.370443    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-maas-api-tls\") pod \"851939a7-2f00-4158-9cfb-9a43641a5c5e\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") "
Nov 02 20:33:20.370477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.370483    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-metrics-certs\") pod \"851939a7-2f00-4158-9cfb-9a43641a5c5e\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") "
Nov 02 20:33:20.370813 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.370524    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/851939a7-2f00-4158-9cfb-9a43641a5c5e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851939a7-2f00-4158-9cfb-9a43641a5c5e-openshift-service-ca\") pod \"851939a7-2f00-4158-9cfb-9a43641a5c5e\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") "
Nov 02 20:33:20.370813 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.370557    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/851939a7-2f00-4158-9cfb-9a43641a5c5e-kube-api-access-xdxbf\" (UniqueName: \"kubernetes.io/projected/851939a7-2f00-4158-9cfb-9a43641a5c5e-kube-api-access-xdxbf\") pod \"851939a7-2f00-4158-9cfb-9a43641a5c5e\" (UID: \"851939a7-2f00-4158-9cfb-9a43641a5c5e\") "
Nov 02 20:33:20.371029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.370994    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851939a7-2f00-4158-9cfb-9a43641a5c5e-openshift-service-ca" pod "851939a7-2f00-4158-9cfb-9a43641a5c5e" (UID: "851939a7-2f00-4158-9cfb-9a43641a5c5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:33:20.373081 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.373044    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-maas-api-tls" pod "851939a7-2f00-4158-9cfb-9a43641a5c5e" (UID: "851939a7-2f00-4158-9cfb-9a43641a5c5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:33:20.373388 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.373354    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-metrics-certs" pod "851939a7-2f00-4158-9cfb-9a43641a5c5e" (UID: "851939a7-2f00-4158-9cfb-9a43641a5c5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:33:20.373961 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.373870    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851939a7-2f00-4158-9cfb-9a43641a5c5e-kube-api-access-xdxbf" pod "851939a7-2f00-4158-9cfb-9a43641a5c5e" (UID: "851939a7-2f00-4158-9cfb-9a43641a5c5e"). InnerVolumeSpecName "kube-api-access-xdxbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:33:20.399814 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.399762    5224 generic.go:356] "Generic (PLEG): container finished" podID="851939a7-2f00-4158-9cfb-9a43641a5c5e" containerID="2b38805d0487dadef5e381a108480079cceacc30c71ca99e3cfb196a09e67c0e" exitCode=0
Nov 02 20:33:20.400019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.399830    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"
Nov 02 20:33:20.400019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.399869    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g" event={"ID":"851939a7-2f00-4158-9cfb-9a43641a5c5e","Type":"ContainerDied","Data":"2b38805d0487dadef5e381a108480079cceacc30c71ca99e3cfb196a09e67c0e"}
Nov 02 20:33:20.400019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.399934    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g" event={"ID":"851939a7-2f00-4158-9cfb-9a43641a5c5e","Type":"ContainerDied","Data":"0f38c6d44f5dcc3921c6ba46214f0b75c53af0f77c967ba9667107691af1fc0d"}
Nov 02 20:33:20.400019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.399970    5224 scope.go:122] "RemoveContainer" containerID="2b38805d0487dadef5e381a108480079cceacc30c71ca99e3cfb196a09e67c0e"
Nov 02 20:33:20.411659 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.411636    5224 scope.go:122] "RemoveContainer" containerID="2b38805d0487dadef5e381a108480079cceacc30c71ca99e3cfb196a09e67c0e"
Nov 02 20:33:20.412077 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:33:20.412014    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b38805d0487dadef5e381a108480079cceacc30c71ca99e3cfb196a09e67c0e\": container with ID starting with 2b38805d0487dadef5e381a108480079cceacc30c71ca99e3cfb196a09e67c0e not found: ID does not exist" containerID="2b38805d0487dadef5e381a108480079cceacc30c71ca99e3cfb196a09e67c0e"
Nov 02 20:33:20.412077 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.412051    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b38805d0487dadef5e381a108480079cceacc30c71ca99e3cfb196a09e67c0e"} err="failed to get container status \"2b38805d0487dadef5e381a108480079cceacc30c71ca99e3cfb196a09e67c0e\": rpc error: code = NotFound desc = could not find container \"2b38805d0487dadef5e381a108480079cceacc30c71ca99e3cfb196a09e67c0e\": container with ID starting with 2b38805d0487dadef5e381a108480079cceacc30c71ca99e3cfb196a09e67c0e not found: ID does not exist"
Nov 02 20:33:20.472682 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.472591    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:20.472682 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.472622    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/851939a7-2f00-4158-9cfb-9a43641a5c5e-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:20.472682 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.472635    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851939a7-2f00-4158-9cfb-9a43641a5c5e-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:20.472682 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.472652    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdxbf\" (UniqueName: \"kubernetes.io/projected/851939a7-2f00-4158-9cfb-9a43641a5c5e-kube-api-access-xdxbf\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:20.565840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.565796    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"]
Nov 02 20:33:20.569624 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:20.569599    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93816f-5d7bbd8d49-7rt5g"]
Nov 02 20:33:21.167358 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:21.167321    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="851939a7-2f00-4158-9cfb-9a43641a5c5e" path="/var/lib/kubelet/pods/851939a7-2f00-4158-9cfb-9a43641a5c5e/volumes"
Nov 02 20:33:21.879570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:21.879530    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:23.352596 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:23.352559    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:33:25.789168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:25.789126    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:25.968342 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:25.968303    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:26.849944 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:26.849900    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:27.360416 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:33:27.360373    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:33:27.461241 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:27.461205    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:27.974532 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:27.974490    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:32.317647 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:32.317613    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:33.618230 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:33.618184    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:37.175744 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:37.175703    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:39.701224 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:39.701184    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:40.654465 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:40.654316    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:40.841437 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:40.841401    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:42.321150 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:42.321082    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:43.429530 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.429482    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"]
Nov 02 20:33:43.430129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430106    5224 cpu_manager.go:414] "containerMap: removing container" podUID="201c4e04-0153-4013-978c-2d8be535a640" containerName="cleanup"
Nov 02 20:33:43.430214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430132    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201c4e04-0153-4013-978c-2d8be535a640" containerName="cleanup"
Nov 02 20:33:43.430214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430155    5224 cpu_manager.go:414] "containerMap: removing container" podUID="3934517c-ae41-4e8e-9651-dda812f7daea" containerName="cleanup"
Nov 02 20:33:43.430214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430165    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3934517c-ae41-4e8e-9651-dda812f7daea" containerName="cleanup"
Nov 02 20:33:43.430214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430179    5224 cpu_manager.go:414] "containerMap: removing container" podUID="0c254164-9b88-49e9-9e90-575daf8b855e" containerName="cleanup"
Nov 02 20:33:43.430214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430187    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c254164-9b88-49e9-9e90-575daf8b855e" containerName="cleanup"
Nov 02 20:33:43.430214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430200    5224 cpu_manager.go:414] "containerMap: removing container" podUID="068b0dec-2759-4b13-ab24-090ee50010de" containerName="cleanup"
Nov 02 20:33:43.430214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430208    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068b0dec-2759-4b13-ab24-090ee50010de" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430223    5224 cpu_manager.go:414] "containerMap: removing container" podUID="1c75ab44-108b-4a6c-8a99-8ee871f8894a" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430230    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c75ab44-108b-4a6c-8a99-8ee871f8894a" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430240    5224 cpu_manager.go:414] "containerMap: removing container" podUID="0c254164-9b88-49e9-9e90-575daf8b855e" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430248    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c254164-9b88-49e9-9e90-575daf8b855e" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430265    5224 cpu_manager.go:414] "containerMap: removing container" podUID="0543f1ee-7bcb-491e-94b0-996ec5133c2f" containerName="maas-api"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430275    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0543f1ee-7bcb-491e-94b0-996ec5133c2f" containerName="maas-api"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430286    5224 cpu_manager.go:414] "containerMap: removing container" podUID="201c4e04-0153-4013-978c-2d8be535a640" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430293    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201c4e04-0153-4013-978c-2d8be535a640" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430309    5224 cpu_manager.go:414] "containerMap: removing container" podUID="201c4e04-0153-4013-978c-2d8be535a640" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430316    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201c4e04-0153-4013-978c-2d8be535a640" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430332    5224 cpu_manager.go:414] "containerMap: removing container" podUID="068b0dec-2759-4b13-ab24-090ee50010de" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430339    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068b0dec-2759-4b13-ab24-090ee50010de" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430352    5224 cpu_manager.go:414] "containerMap: removing container" podUID="b2003aed-acbb-4307-9733-fd6e0b791217" containerName="maas-api"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430360    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2003aed-acbb-4307-9733-fd6e0b791217" containerName="maas-api"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430373    5224 cpu_manager.go:414] "containerMap: removing container" podUID="1c75ab44-108b-4a6c-8a99-8ee871f8894a" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430380    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c75ab44-108b-4a6c-8a99-8ee871f8894a" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430399    5224 cpu_manager.go:414] "containerMap: removing container" podUID="3934517c-ae41-4e8e-9651-dda812f7daea" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430406    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3934517c-ae41-4e8e-9651-dda812f7daea" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430416    5224 cpu_manager.go:414] "containerMap: removing container" podUID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" containerName="istio-proxy"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430423    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" containerName="istio-proxy"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430446    5224 cpu_manager.go:414] "containerMap: removing container" podUID="851939a7-2f00-4158-9cfb-9a43641a5c5e" containerName="maas-api"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430455    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="851939a7-2f00-4158-9cfb-9a43641a5c5e" containerName="maas-api"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430466    5224 cpu_manager.go:414] "containerMap: removing container" podUID="1c75ab44-108b-4a6c-8a99-8ee871f8894a" containerName="cleanup"
Nov 02 20:33:43.430578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430473    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c75ab44-108b-4a6c-8a99-8ee871f8894a" containerName="cleanup"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430595    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3934517c-ae41-4e8e-9651-dda812f7daea" containerName="cleanup"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430606    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c75ab44-108b-4a6c-8a99-8ee871f8894a" containerName="cleanup"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430619    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c802b4b-1dec-400b-a1e7-5507933ef8e3" containerName="istio-proxy"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430627    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c254164-9b88-49e9-9e90-575daf8b855e" containerName="cleanup"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430638    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="851939a7-2f00-4158-9cfb-9a43641a5c5e" containerName="maas-api"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430649    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="068b0dec-2759-4b13-ab24-090ee50010de" containerName="cleanup"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430657    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c75ab44-108b-4a6c-8a99-8ee871f8894a" containerName="cleanup"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430668    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="201c4e04-0153-4013-978c-2d8be535a640" containerName="cleanup"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430677    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3934517c-ae41-4e8e-9651-dda812f7daea" containerName="cleanup"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430686    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="068b0dec-2759-4b13-ab24-090ee50010de" containerName="cleanup"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430694    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c254164-9b88-49e9-9e90-575daf8b855e" containerName="cleanup"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430704    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0543f1ee-7bcb-491e-94b0-996ec5133c2f" containerName="maas-api"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430712    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="201c4e04-0153-4013-978c-2d8be535a640" containerName="cleanup"
Nov 02 20:33:43.431753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.430722    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2003aed-acbb-4307-9733-fd6e0b791217" containerName="maas-api"
Nov 02 20:33:43.435262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.435238    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"
Nov 02 20:33:43.484722 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.484689    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"]
Nov 02 20:33:43.488597 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.488570    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:33:43.488823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.488803    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:33:43.533937 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.533898    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:33:43.533937 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.533920    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:33:43.626171 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.626128    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkjsf\" (UniqueName: \"kubernetes.io/projected/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-kube-api-access-rkjsf\") pod \"maas-api-revoke-sub-4ea1fe76-tmvp5\" (UID: \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"
Nov 02 20:33:43.626367 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.626296    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ea1fe76-tmvp5\" (UID: \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"
Nov 02 20:33:43.626367 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.626358    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ea1fe76-tmvp5\" (UID: \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"
Nov 02 20:33:43.727134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.727035    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ea1fe76-tmvp5\" (UID: \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"
Nov 02 20:33:43.727134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.727118    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ea1fe76-tmvp5\" (UID: \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"
Nov 02 20:33:43.727412 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.727162    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkjsf\" (UniqueName: \"kubernetes.io/projected/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-kube-api-access-rkjsf\") pod \"maas-api-revoke-sub-4ea1fe76-tmvp5\" (UID: \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"
Nov 02 20:33:43.727855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.727824    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ea1fe76-tmvp5\" (UID: \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"
Nov 02 20:33:43.729559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.729537    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ea1fe76-tmvp5\" (UID: \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"
Nov 02 20:33:43.785320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:43.785285    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkjsf\" (UniqueName: \"kubernetes.io/projected/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-kube-api-access-rkjsf\") pod \"maas-api-revoke-sub-4ea1fe76-tmvp5\" (UID: \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"
Nov 02 20:33:44.048464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:44.048430    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"
Nov 02 20:33:44.414050 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:33:44.414012    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d773d4_a9a5_43ca_b88f_8b0e3c303e2a.slice/crio-fd3ce02851b7701bd1a8982fc39db77dc841e7969b3f424e324263ec717448e4 WatchSource:0}: Error finding container fd3ce02851b7701bd1a8982fc39db77dc841e7969b3f424e324263ec717448e4: Status 404 returned error can't find the container with id fd3ce02851b7701bd1a8982fc39db77dc841e7969b3f424e324263ec717448e4
Nov 02 20:33:44.416527 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:44.416498    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"]
Nov 02 20:33:44.543002 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:44.542964    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5" event={"ID":"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a","Type":"ContainerStarted","Data":"fd3ce02851b7701bd1a8982fc39db77dc841e7969b3f424e324263ec717448e4"}
Nov 02 20:33:44.631356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:44.631314    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:45.055625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.055591    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"]
Nov 02 20:33:45.056047 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.056033    5224 cpu_manager.go:414] "containerMap: removing container" podUID="3934517c-ae41-4e8e-9651-dda812f7daea" containerName="cleanup"
Nov 02 20:33:45.056104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.056049    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3934517c-ae41-4e8e-9651-dda812f7daea" containerName="cleanup"
Nov 02 20:33:45.056104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.056083    5224 cpu_manager.go:414] "containerMap: removing container" podUID="0c254164-9b88-49e9-9e90-575daf8b855e" containerName="cleanup"
Nov 02 20:33:45.056104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.056089    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c254164-9b88-49e9-9e90-575daf8b855e" containerName="cleanup"
Nov 02 20:33:45.056207 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.056107    5224 cpu_manager.go:414] "containerMap: removing container" podUID="068b0dec-2759-4b13-ab24-090ee50010de" containerName="cleanup"
Nov 02 20:33:45.056207 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.056112    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068b0dec-2759-4b13-ab24-090ee50010de" containerName="cleanup"
Nov 02 20:33:45.056207 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.056184    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="068b0dec-2759-4b13-ab24-090ee50010de" containerName="cleanup"
Nov 02 20:33:45.056207 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.056195    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="201c4e04-0153-4013-978c-2d8be535a640" containerName="cleanup"
Nov 02 20:33:45.056207 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.056203    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3934517c-ae41-4e8e-9651-dda812f7daea" containerName="cleanup"
Nov 02 20:33:45.056374 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.056209    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c254164-9b88-49e9-9e90-575daf8b855e" containerName="cleanup"
Nov 02 20:33:45.056374 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.056215    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c75ab44-108b-4a6c-8a99-8ee871f8894a" containerName="cleanup"
Nov 02 20:33:45.059430 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.059411    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.116038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.116005    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"]
Nov 02 20:33:45.139686 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.139638    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlm64\" (UniqueName: \"kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-kube-api-access-mlm64\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.139912 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.139795    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-certs\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.139912 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.139841    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4880ac-7310-4d51-86f2-75634f425b28-istiod-ca-cert\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.139912 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.139899    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-envoy\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.140049 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.139942    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-data\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.140049 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.140001    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-socket\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.140145 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.140128    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-credential-socket\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.140193 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.140157    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4880ac-7310-4d51-86f2-75634f425b28-istio-podinfo\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.140193 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.140185    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-istio-token\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.185869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.185829    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-2c6ce9-openshift-default-dockercfg-zkxks\""
Nov 02 20:33:45.216913 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.216876    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:45.241581 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.241524    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-credential-socket\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.241998 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.241962    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4880ac-7310-4d51-86f2-75634f425b28-istio-podinfo\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.242178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.242154    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-istio-token\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.242307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.242210    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlm64\" (UniqueName: \"kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-kube-api-access-mlm64\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.242307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.242242    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-certs\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.242307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.242284    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4880ac-7310-4d51-86f2-75634f425b28-istiod-ca-cert\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.242487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.242451    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-envoy\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.242546 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.242500    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-data\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.242604 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.242567    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-socket\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.242678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.242021    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-credential-socket\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.243007 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.242979    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-certs\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.243092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.243020    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-socket\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.243092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.243080    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-data\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.243301 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.243276    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4880ac-7310-4d51-86f2-75634f425b28-istiod-ca-cert\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.245707 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.245684    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-envoy\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.245916 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.245897    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4880ac-7310-4d51-86f2-75634f425b28-istio-podinfo\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.301968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.301920    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlm64\" (UniqueName: \"kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-kube-api-access-mlm64\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.302436 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.302409    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-istio-token\") pod \"e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") " pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.374415 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.374378    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:45.395462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.395399    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:45.550954 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.550917    5224 generic.go:356] "Generic (PLEG): container finished" podID="85d773d4-a9a5-43ca-b88f-8b0e3c303e2a" containerID="0741bfdc8b687d1e751b7b9c52a620d52122139299e30d1b3d7503221c5e83e3" exitCode=0
Nov 02 20:33:45.551354 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.551000    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5" event={"ID":"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a","Type":"ContainerDied","Data":"0741bfdc8b687d1e751b7b9c52a620d52122139299e30d1b3d7503221c5e83e3"}
Nov 02 20:33:45.739715 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.739597    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:45.818456 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:33:45.818411    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4880ac_7310_4d51_86f2_75634f425b28.slice/crio-a0f2a65597c8ba5e53282550a5f31618ae8a1818cf18aa1e48ec43c019148b2b WatchSource:0}: Error finding container a0f2a65597c8ba5e53282550a5f31618ae8a1818cf18aa1e48ec43c019148b2b: Status 404 returned error can't find the container with id a0f2a65597c8ba5e53282550a5f31618ae8a1818cf18aa1e48ec43c019148b2b
Nov 02 20:33:45.819930 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.819898    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"]
Nov 02 20:33:45.821433 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.821387    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:33:45.821534 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.821492    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:33:45.821572 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:45.821542    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:33:46.559323 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:46.559278    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh" event={"ID":"4b4880ac-7310-4d51-86f2-75634f425b28","Type":"ContainerStarted","Data":"ae092d241cf7cfcbe00d35b7886d7400707e30c0f02e2202cb31ae47ad91076e"}
Nov 02 20:33:46.559848 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:46.559345    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh" event={"ID":"4b4880ac-7310-4d51-86f2-75634f425b28","Type":"ContainerStarted","Data":"a0f2a65597c8ba5e53282550a5f31618ae8a1818cf18aa1e48ec43c019148b2b"}
Nov 02 20:33:46.679303 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:46.679237    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh" podStartSLOduration=1.679221563 podStartE2EDuration="1.679221563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:46.677608548 +0000 UTC m=+2707.675297157" watchObservedRunningTime="2026-11-02 20:33:46.679221563 +0000 UTC m=+2707.676910172"
Nov 02 20:33:46.711721 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:46.711691    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"
Nov 02 20:33:46.760213 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:46.760174    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-kube-api-access-rkjsf\" (UniqueName: \"kubernetes.io/projected/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-kube-api-access-rkjsf\") pod \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\" (UID: \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\") "
Nov 02 20:33:46.760213 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:46.760213    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-openshift-ca\") pod \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\" (UID: \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\") "
Nov 02 20:33:46.760433 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:46.760256    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-xks-ca\") pod \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\" (UID: \"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a\") "
Nov 02 20:33:46.760588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:46.760568    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-openshift-ca" pod "85d773d4-a9a5-43ca-b88f-8b0e3c303e2a" (UID: "85d773d4-a9a5-43ca-b88f-8b0e3c303e2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:33:46.760819 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:46.760757    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:46.762356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:46.762330    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-xks-ca" pod "85d773d4-a9a5-43ca-b88f-8b0e3c303e2a" (UID: "85d773d4-a9a5-43ca-b88f-8b0e3c303e2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:33:46.762482 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:46.762402    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-kube-api-access-rkjsf" pod "85d773d4-a9a5-43ca-b88f-8b0e3c303e2a" (UID: "85d773d4-a9a5-43ca-b88f-8b0e3c303e2a"). InnerVolumeSpecName "kube-api-access-rkjsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:33:46.861740 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:46.861654    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkjsf\" (UniqueName: \"kubernetes.io/projected/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-kube-api-access-rkjsf\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:46.861740 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:46.861686    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:47.286740 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:47.286700    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:47.375261 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:47.375223    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:47.380863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:47.380829    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:47.566454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:47.566411    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5" event={"ID":"85d773d4-a9a5-43ca-b88f-8b0e3c303e2a","Type":"ContainerDied","Data":"fd3ce02851b7701bd1a8982fc39db77dc841e7969b3f424e324263ec717448e4"}
Nov 02 20:33:47.566454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:47.566453    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"
Nov 02 20:33:47.567060 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:47.566469    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd3ce02851b7701bd1a8982fc39db77dc841e7969b3f424e324263ec717448e4"
Nov 02 20:33:47.567060 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:47.566688    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:47.567982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:47.567959    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:33:49.375228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:49.375108    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:49.570216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:49.570179    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:49.753024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:49.752907    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:52.457337 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:52.457284    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:52.640569 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:52.640522    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:52.845096 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:52.845050    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:55.035500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:55.035456    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:55.674212 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:55.674176    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:55.928643 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:55.928538    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:57.360830 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:33:57.360795    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:33:57.455968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:57.455925    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:59.352890 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:59.352850    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:59.609717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:59.609623    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:33:59.793361 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:33:59.793321    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:00.019329 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:00.019288    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:00.704093 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:00.704004    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:02.515091 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:02.515051    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:02.744226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:02.744184    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:02.946822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:02.946710    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:07.689820 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:07.689780    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"]
Nov 02 20:34:07.690261 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:07.689993    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" podUID="a9bd828b-4265-447e-89da-c94e97503b89" containerName="istio-proxy" containerID="cri-o://875af3bd659c015bd90814f346426da7b1f1aace5830778b9dde91fabe57df24" gracePeriod=30
Nov 02 20:34:08.373036 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:08.372997    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:11.138010 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.137964    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:11.293627 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.293578    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"]
Nov 02 20:34:11.294083 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.294068    5224 cpu_manager.go:414] "containerMap: removing container" podUID="85d773d4-a9a5-43ca-b88f-8b0e3c303e2a" containerName="revoke-keys"
Nov 02 20:34:11.294140 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.294084    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d773d4-a9a5-43ca-b88f-8b0e3c303e2a" containerName="revoke-keys"
Nov 02 20:34:11.294177 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.294163    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d773d4-a9a5-43ca-b88f-8b0e3c303e2a" containerName="revoke-keys"
Nov 02 20:34:11.297110 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.297094    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"
Nov 02 20:34:11.342002 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.341944    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39016714-nlbt9\" (UID: \"492ba5e8-9ccc-4226-805d-97a3174b27a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"
Nov 02 20:34:11.342254 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.342026    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2swd5\" (UniqueName: \"kubernetes.io/projected/492ba5e8-9ccc-4226-805d-97a3174b27a0-kube-api-access-2swd5\") pod \"maas-api-revoke-sub-39016714-nlbt9\" (UID: \"492ba5e8-9ccc-4226-805d-97a3174b27a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"
Nov 02 20:34:11.342254 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.342176    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39016714-nlbt9\" (UID: \"492ba5e8-9ccc-4226-805d-97a3174b27a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"
Nov 02 20:34:11.349741 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.349707    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"]
Nov 02 20:34:11.369320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.369292    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:34:11.398642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.398562    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:34:11.398642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.398565    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:34:11.398642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.398630    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:34:11.443168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.443129    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39016714-nlbt9\" (UID: \"492ba5e8-9ccc-4226-805d-97a3174b27a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"
Nov 02 20:34:11.443365 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.443242    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39016714-nlbt9\" (UID: \"492ba5e8-9ccc-4226-805d-97a3174b27a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"
Nov 02 20:34:11.443365 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.443278    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2swd5\" (UniqueName: \"kubernetes.io/projected/492ba5e8-9ccc-4226-805d-97a3174b27a0-kube-api-access-2swd5\") pod \"maas-api-revoke-sub-39016714-nlbt9\" (UID: \"492ba5e8-9ccc-4226-805d-97a3174b27a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"
Nov 02 20:34:11.443953 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.443933    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39016714-nlbt9\" (UID: \"492ba5e8-9ccc-4226-805d-97a3174b27a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"
Nov 02 20:34:11.445643 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.445617    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39016714-nlbt9\" (UID: \"492ba5e8-9ccc-4226-805d-97a3174b27a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"
Nov 02 20:34:11.501220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.501187    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2swd5\" (UniqueName: \"kubernetes.io/projected/492ba5e8-9ccc-4226-805d-97a3174b27a0-kube-api-access-2swd5\") pod \"maas-api-revoke-sub-39016714-nlbt9\" (UID: \"492ba5e8-9ccc-4226-805d-97a3174b27a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"
Nov 02 20:34:11.610444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.610404    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"
Nov 02 20:34:11.977719 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:34:11.977676    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492ba5e8_9ccc_4226_805d_97a3174b27a0.slice/crio-4dc80a2a261f3020c75b098c8373df2c96501fe2173db7478d3efa6e62b30c12 WatchSource:0}: Error finding container 4dc80a2a261f3020c75b098c8373df2c96501fe2173db7478d3efa6e62b30c12: Status 404 returned error can't find the container with id 4dc80a2a261f3020c75b098c8373df2c96501fe2173db7478d3efa6e62b30c12
Nov 02 20:34:11.980419 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:11.980383    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"]
Nov 02 20:34:12.235955 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.235867    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"]
Nov 02 20:34:12.240558 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.240539    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.250281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.250253    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.250416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.250289    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.250416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.250326    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq8wc\" (UniqueName: \"kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-kube-api-access-tq8wc\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.250541 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.250413    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.250634 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.250614    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.250684 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.250665    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.250809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.250791    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.250905 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.250893    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.250969 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.250958    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.266390 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.266362    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"]
Nov 02 20:34:12.270000 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.269977    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.333441 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.333403    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"]
Nov 02 20:34:12.351865 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.351830    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.352063 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.351874    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.352063 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.351966    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.352063 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352000    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68pqp\" (UniqueName: \"kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-kube-api-access-68pqp\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.352063 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352047    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.352311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352093    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.352311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352180    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.352311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352217    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.352311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352256    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq8wc\" (UniqueName: \"kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-kube-api-access-tq8wc\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.352311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352287    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.352827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352367    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.352827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352396    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.352827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352433    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.352827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352461    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.352827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352547    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.352827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352598    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.352827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352668    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.352827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352694    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.352827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352737    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.352827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352758    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.353369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.352886    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.353369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.353083    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.354042 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.354015    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.355203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.355174    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.355465 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.355447    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.384512 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.384474    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"]
Nov 02 20:34:12.410728 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.410689    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.410907 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.410869    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq8wc\" (UniqueName: \"kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-kube-api-access-tq8wc\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.454228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.454183    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.454414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.454258    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.454414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.454304    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.454414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.454341    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.454595 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.454435    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.454595 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.454460    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.454595 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.454494    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68pqp\" (UniqueName: \"kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-kube-api-access-68pqp\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.454595 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.454538    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.454851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.454596    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.454851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.454684    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.454851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.454731    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.455029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.455008    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.455150 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.455128    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.455307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.455286    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.457191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.457159    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.457351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.457335    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.512071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.511985    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.512235 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.512071    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68pqp\" (UniqueName: \"kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-kube-api-access-68pqp\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.555094 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.555053    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:12.583528 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.583489    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:12.722945 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.722896    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9" event={"ID":"492ba5e8-9ccc-4226-805d-97a3174b27a0","Type":"ContainerStarted","Data":"9d4b83f5010bf56584d154a5beef9a4a90a361dda5a92cdd7e695cc379d00061"}
Nov 02 20:34:12.723114 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.723007    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9" event={"ID":"492ba5e8-9ccc-4226-805d-97a3174b27a0","Type":"ContainerStarted","Data":"4dc80a2a261f3020c75b098c8373df2c96501fe2173db7478d3efa6e62b30c12"}
Nov 02 20:34:12.839260 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.839204    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9" podStartSLOduration=1.8391821259999999 podStartE2EDuration="1.839182126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:12.837590046 +0000 UTC m=+2733.835278666" watchObservedRunningTime="2026-11-02 20:34:12.839182126 +0000 UTC m=+2733.836870786"
Nov 02 20:34:12.962910 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.962882    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 20:34:12.983087 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.982843    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"]
Nov 02 20:34:12.984450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.983757    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:12.984450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.983860    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:12.984450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.983943    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:12.985244 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.985206    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:12.985324 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.985274    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:12.985324 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.985304    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:12.986373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:12.986345    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"]
Nov 02 20:34:13.046390 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.046347    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:13.064869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.063616    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9bd828b-4265-447e-89da-c94e97503b89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bd828b-4265-447e-89da-c94e97503b89-istio-podinfo\") pod \"a9bd828b-4265-447e-89da-c94e97503b89\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") "
Nov 02 20:34:13.064869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.063679    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-data\") pod \"a9bd828b-4265-447e-89da-c94e97503b89\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") "
Nov 02 20:34:13.064869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.063725    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9bd828b-4265-447e-89da-c94e97503b89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bd828b-4265-447e-89da-c94e97503b89-istiod-ca-cert\") pod \"a9bd828b-4265-447e-89da-c94e97503b89\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") "
Nov 02 20:34:13.064869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.063831    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-envoy\") pod \"a9bd828b-4265-447e-89da-c94e97503b89\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") "
Nov 02 20:34:13.064869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.063876    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-credential-socket\") pod \"a9bd828b-4265-447e-89da-c94e97503b89\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") "
Nov 02 20:34:13.064869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.063918    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-istio-token\" (UniqueName: \"kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-istio-token\") pod \"a9bd828b-4265-447e-89da-c94e97503b89\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") "
Nov 02 20:34:13.064869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.063958    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-kube-api-access-qdnlb\" (UniqueName: \"kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-kube-api-access-qdnlb\") pod \"a9bd828b-4265-447e-89da-c94e97503b89\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") "
Nov 02 20:34:13.064869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.064030    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-socket\") pod \"a9bd828b-4265-447e-89da-c94e97503b89\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") "
Nov 02 20:34:13.064869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.064091    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-certs\") pod \"a9bd828b-4265-447e-89da-c94e97503b89\" (UID: \"a9bd828b-4265-447e-89da-c94e97503b89\") "
Nov 02 20:34:13.065903 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.065853    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-data" pod "a9bd828b-4265-447e-89da-c94e97503b89" (UID: "a9bd828b-4265-447e-89da-c94e97503b89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:13.066797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.066103    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-certs" pod "a9bd828b-4265-447e-89da-c94e97503b89" (UID: "a9bd828b-4265-447e-89da-c94e97503b89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:13.067083 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.066962    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-socket" pod "a9bd828b-4265-447e-89da-c94e97503b89" (UID: "a9bd828b-4265-447e-89da-c94e97503b89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:13.067379 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.067325    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9bd828b-4265-447e-89da-c94e97503b89-istiod-ca-cert" pod "a9bd828b-4265-447e-89da-c94e97503b89" (UID: "a9bd828b-4265-447e-89da-c94e97503b89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:34:13.069144 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.069105    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-credential-socket" pod "a9bd828b-4265-447e-89da-c94e97503b89" (UID: "a9bd828b-4265-447e-89da-c94e97503b89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:13.072444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.072415    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9bd828b-4265-447e-89da-c94e97503b89-istio-podinfo" pod "a9bd828b-4265-447e-89da-c94e97503b89" (UID: "a9bd828b-4265-447e-89da-c94e97503b89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:34:13.072444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.072429    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-envoy" pod "a9bd828b-4265-447e-89da-c94e97503b89" (UID: "a9bd828b-4265-447e-89da-c94e97503b89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:13.073415 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.073386    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-istio-token" pod "a9bd828b-4265-447e-89da-c94e97503b89" (UID: "a9bd828b-4265-447e-89da-c94e97503b89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:34:13.074923 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.074898    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-kube-api-access-qdnlb" pod "a9bd828b-4265-447e-89da-c94e97503b89" (UID: "a9bd828b-4265-447e-89da-c94e97503b89"). InnerVolumeSpecName "kube-api-access-qdnlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:34:13.166872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.165844    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bd828b-4265-447e-89da-c94e97503b89-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:13.166872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.165893    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:13.166872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.165908    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:13.166872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.165921    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:13.166872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.165949    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdnlb\" (UniqueName: \"kubernetes.io/projected/a9bd828b-4265-447e-89da-c94e97503b89-kube-api-access-qdnlb\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:13.166872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.165965    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:13.166872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.165979    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:13.166872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.165991    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bd828b-4265-447e-89da-c94e97503b89-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:13.166872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.166017    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bd828b-4265-447e-89da-c94e97503b89-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:13.734576 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.734448    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9" event={"ID":"23ca0a07-1d6a-49d1-9f58-aeef040acc4d","Type":"ContainerStarted","Data":"792962527b7abdef3dd3d60eb59561cabf0a9e19d53b1ab51b6da3456849e246"}
Nov 02 20:34:13.734576 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.734529    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9" event={"ID":"23ca0a07-1d6a-49d1-9f58-aeef040acc4d","Type":"ContainerStarted","Data":"102e90b314aaa500e9b55bf4486cadb042c7e66bcdc423a4ca05aafa8c7fb1a2"}
Nov 02 20:34:13.737024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.736993    5224 generic.go:356] "Generic (PLEG): container finished" podID="492ba5e8-9ccc-4226-805d-97a3174b27a0" containerID="9d4b83f5010bf56584d154a5beef9a4a90a361dda5a92cdd7e695cc379d00061" exitCode=0
Nov 02 20:34:13.737195 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.737028    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9" event={"ID":"492ba5e8-9ccc-4226-805d-97a3174b27a0","Type":"ContainerDied","Data":"9d4b83f5010bf56584d154a5beef9a4a90a361dda5a92cdd7e695cc379d00061"}
Nov 02 20:34:13.738563 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.738542    5224 generic.go:356] "Generic (PLEG): container finished" podID="a9bd828b-4265-447e-89da-c94e97503b89" containerID="875af3bd659c015bd90814f346426da7b1f1aace5830778b9dde91fabe57df24" exitCode=0
Nov 02 20:34:13.738682 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.738580    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"
Nov 02 20:34:13.738682 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.738619    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" event={"ID":"a9bd828b-4265-447e-89da-c94e97503b89","Type":"ContainerDied","Data":"875af3bd659c015bd90814f346426da7b1f1aace5830778b9dde91fabe57df24"}
Nov 02 20:34:13.738682 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.738665    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv" event={"ID":"a9bd828b-4265-447e-89da-c94e97503b89","Type":"ContainerDied","Data":"ea62bbb199734f03ce30cc935ebe26c471cf822822245da61f59f96454b637d9"}
Nov 02 20:34:13.738796 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.738697    5224 scope.go:122] "RemoveContainer" containerID="875af3bd659c015bd90814f346426da7b1f1aace5830778b9dde91fabe57df24"
Nov 02 20:34:13.740454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.740432    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79" event={"ID":"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1","Type":"ContainerStarted","Data":"dea8a2cf59b74015165d659e97e14b493541b8b71260da3a11cc5a756610b5ec"}
Nov 02 20:34:13.740560 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.740469    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79" event={"ID":"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1","Type":"ContainerStarted","Data":"ad094459614b77a3545e12c77f645f8cb054d0c33dcca897bcb35d1bad82d3a1"}
Nov 02 20:34:13.748708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.748680    5224 scope.go:122] "RemoveContainer" containerID="875af3bd659c015bd90814f346426da7b1f1aace5830778b9dde91fabe57df24"
Nov 02 20:34:13.749024 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:34:13.749006    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875af3bd659c015bd90814f346426da7b1f1aace5830778b9dde91fabe57df24\": container with ID starting with 875af3bd659c015bd90814f346426da7b1f1aace5830778b9dde91fabe57df24 not found: ID does not exist" containerID="875af3bd659c015bd90814f346426da7b1f1aace5830778b9dde91fabe57df24"
Nov 02 20:34:13.749106 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.749039    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875af3bd659c015bd90814f346426da7b1f1aace5830778b9dde91fabe57df24"} err="failed to get container status \"875af3bd659c015bd90814f346426da7b1f1aace5830778b9dde91fabe57df24\": rpc error: code = NotFound desc = could not find container \"875af3bd659c015bd90814f346426da7b1f1aace5830778b9dde91fabe57df24\": container with ID starting with 875af3bd659c015bd90814f346426da7b1f1aace5830778b9dde91fabe57df24 not found: ID does not exist"
Nov 02 20:34:13.854834 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:13.854684    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9" podStartSLOduration=1.8546645499999999 podStartE2EDuration="1.85466455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:13.853409015 +0000 UTC m=+2734.851097626" watchObservedRunningTime="2026-11-02 20:34:13.85466455 +0000 UTC m=+2734.852353163"
Nov 02 20:34:14.111641 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:14.110159    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79" podStartSLOduration=2.110140303 podStartE2EDuration="2.110140303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:13.977355562 +0000 UTC m=+2734.975044178" watchObservedRunningTime="2026-11-02 20:34:14.110140303 +0000 UTC m=+2735.107828913"
Nov 02 20:34:14.111998 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:14.111969    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"]
Nov 02 20:34:14.115307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:14.114244    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9n4hv"]
Nov 02 20:34:14.560793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:14.559484    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:14.560793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:14.559620    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 20:34:14.560793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:14.559673    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79" podUID="e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 20:34:14.585794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:14.585500    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:14.585794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:14.585651    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 20:34:14.585794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:14.585700    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9" podUID="23ca0a07-1d6a-49d1-9f58-aeef040acc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 20:34:15.082796 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.082004    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"
Nov 02 20:34:15.135857 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.135820    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-xks-ca\") pod \"492ba5e8-9ccc-4226-805d-97a3174b27a0\" (UID: \"492ba5e8-9ccc-4226-805d-97a3174b27a0\") "
Nov 02 20:34:15.136056 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.135893    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-openshift-ca\") pod \"492ba5e8-9ccc-4226-805d-97a3174b27a0\" (UID: \"492ba5e8-9ccc-4226-805d-97a3174b27a0\") "
Nov 02 20:34:15.136056 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.135976    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492ba5e8-9ccc-4226-805d-97a3174b27a0-kube-api-access-2swd5\" (UniqueName: \"kubernetes.io/projected/492ba5e8-9ccc-4226-805d-97a3174b27a0-kube-api-access-2swd5\") pod \"492ba5e8-9ccc-4226-805d-97a3174b27a0\" (UID: \"492ba5e8-9ccc-4226-805d-97a3174b27a0\") "
Nov 02 20:34:15.136628 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.136604    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-openshift-ca" pod "492ba5e8-9ccc-4226-805d-97a3174b27a0" (UID: "492ba5e8-9ccc-4226-805d-97a3174b27a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:34:15.138107 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.138079    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-xks-ca" pod "492ba5e8-9ccc-4226-805d-97a3174b27a0" (UID: "492ba5e8-9ccc-4226-805d-97a3174b27a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:34:15.138382 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.138365    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492ba5e8-9ccc-4226-805d-97a3174b27a0-kube-api-access-2swd5" pod "492ba5e8-9ccc-4226-805d-97a3174b27a0" (UID: "492ba5e8-9ccc-4226-805d-97a3174b27a0"). InnerVolumeSpecName "kube-api-access-2swd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:34:15.209675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.209589    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9bd828b-4265-447e-89da-c94e97503b89" path="/var/lib/kubelet/pods/a9bd828b-4265-447e-89da-c94e97503b89/volumes"
Nov 02 20:34:15.227577 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.227547    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:15.237995 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.237967    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:15.237995 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.237996    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2swd5\" (UniqueName: \"kubernetes.io/projected/492ba5e8-9ccc-4226-805d-97a3174b27a0-kube-api-access-2swd5\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:15.238226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.238013    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492ba5e8-9ccc-4226-805d-97a3174b27a0-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:15.401291 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.401236    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:15.561795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.559305    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 20:34:15.561795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.559374    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79" podUID="e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 20:34:15.588320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.588272    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 20:34:15.588642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.588612    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9" podUID="23ca0a07-1d6a-49d1-9f58-aeef040acc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 20:34:15.780871 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.780832    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9" event={"ID":"492ba5e8-9ccc-4226-805d-97a3174b27a0","Type":"ContainerDied","Data":"4dc80a2a261f3020c75b098c8373df2c96501fe2173db7478d3efa6e62b30c12"}
Nov 02 20:34:15.781195 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.781176    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dc80a2a261f3020c75b098c8373df2c96501fe2173db7478d3efa6e62b30c12"
Nov 02 20:34:15.781346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:15.781335    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"
Nov 02 20:34:16.558795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.557196    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 20:34:16.558795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.557272    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79" podUID="e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 20:34:16.588809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.587320    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 20:34:16.588809 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.587391    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9" podUID="23ca0a07-1d6a-49d1-9f58-aeef040acc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 20:34:16.771794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.769855    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"]
Nov 02 20:34:16.771794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.770546    5224 cpu_manager.go:414] "containerMap: removing container" podUID="492ba5e8-9ccc-4226-805d-97a3174b27a0" containerName="revoke-keys"
Nov 02 20:34:16.771794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.770568    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492ba5e8-9ccc-4226-805d-97a3174b27a0" containerName="revoke-keys"
Nov 02 20:34:16.771794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.770601    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a9bd828b-4265-447e-89da-c94e97503b89" containerName="istio-proxy"
Nov 02 20:34:16.771794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.770609    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9bd828b-4265-447e-89da-c94e97503b89" containerName="istio-proxy"
Nov 02 20:34:16.771794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.770783    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="492ba5e8-9ccc-4226-805d-97a3174b27a0" containerName="revoke-keys"
Nov 02 20:34:16.771794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.770804    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9bd828b-4265-447e-89da-c94e97503b89" containerName="istio-proxy"
Nov 02 20:34:16.775811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.775599    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:16.808013 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.807975    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"]
Nov 02 20:34:16.811858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.811805    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:16.838473 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.838440    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-14d090-openshift-default-dockercfg-5wl9l\""
Nov 02 20:34:16.878274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.878228    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"]
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.883989    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884043    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884077    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884119    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884161    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884211    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884234    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884271    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884299    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884326    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884362    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884416    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884445    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884480    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884522    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884573    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884642    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp76f\" (UniqueName: \"kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-kube-api-access-sp76f\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:16.885792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.884698    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km6nb\" (UniqueName: \"kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-kube-api-access-km6nb\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:16.913842 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.913801    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"]
Nov 02 20:34:16.928460 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.928425    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:16.986518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.986455    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:16.987660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.987629    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:16.994530 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.994491    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.999960    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000010    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000035    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000071    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000123    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000151    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000183    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000232    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000287    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000368    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp76f\" (UniqueName: \"kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-kube-api-access-sp76f\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000427    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km6nb\" (UniqueName: \"kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-kube-api-access-km6nb\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000488    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000519    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000552    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.000603    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.001042    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.001316    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.001513    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.001955    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.988225    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:16.995120    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.002888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.002449    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.004164 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.003251    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.004164 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.003265    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.004164 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.003846    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.004389 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.004234    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.006439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.006417    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.006578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.006107    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.007987 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.007941    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.079847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.078778    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km6nb\" (UniqueName: \"kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-kube-api-access-km6nb\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.079847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.079303    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.104432 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.102358    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp76f\" (UniqueName: \"kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-kube-api-access-sp76f\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.104432 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.102839    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.108924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.108890    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:17.126028 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.125823    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:17.196794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.196556    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:17.568794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.557244    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 20:34:17.568794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.557318    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79" podUID="e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 20:34:17.594075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.585884    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 20:34:17.594075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.585956    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9" podUID="23ca0a07-1d6a-49d1-9f58-aeef040acc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 20:34:17.612687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.612652    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"]
Nov 02 20:34:17.613999 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:34:17.613968    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f7c74c_68c9_4da9_b5cc_07b37f6ac76b.slice/crio-abb26d0f88291e34c4b34067c412db296a9b19a7206f3c0cf65304fd33b3abf8 WatchSource:0}: Error finding container abb26d0f88291e34c4b34067c412db296a9b19a7206f3c0cf65304fd33b3abf8: Status 404 returned error can't find the container with id abb26d0f88291e34c4b34067c412db296a9b19a7206f3c0cf65304fd33b3abf8
Nov 02 20:34:17.616607 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.616573    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:17.616720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.616659    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:17.616720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.616704    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:17.670197 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:34:17.661872    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3317681e_cd5b_4c6f_a8cb_a3dec259764e.slice/crio-08376eac095c9337fd55c77d4ae14a43d95e5ad9aaf25d7672e46afafbb8a433 WatchSource:0}: Error finding container 08376eac095c9337fd55c77d4ae14a43d95e5ad9aaf25d7672e46afafbb8a433: Status 404 returned error can't find the container with id 08376eac095c9337fd55c77d4ae14a43d95e5ad9aaf25d7672e46afafbb8a433
Nov 02 20:34:17.670197 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.665570    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"]
Nov 02 20:34:17.677038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.674383    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:17.677038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.674482    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:17.677038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.674525    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:17.825220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.825094    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg" event={"ID":"3317681e-cd5b-4c6f-a8cb-a3dec259764e","Type":"ContainerStarted","Data":"08376eac095c9337fd55c77d4ae14a43d95e5ad9aaf25d7672e46afafbb8a433"}
Nov 02 20:34:17.829100 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.828428    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" event={"ID":"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b","Type":"ContainerStarted","Data":"06a53efd967dd0ce78402bb7877c1442d9d01b3427c4129354ca2bab1111b150"}
Nov 02 20:34:17.829100 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.828473    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" event={"ID":"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b","Type":"ContainerStarted","Data":"abb26d0f88291e34c4b34067c412db296a9b19a7206f3c0cf65304fd33b3abf8"}
Nov 02 20:34:17.842801 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.841403    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:17.980794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.980132    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:34:18.002793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:17.996789    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" podStartSLOduration=1.996751165 podStartE2EDuration="1.996751165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:17.964807136 +0000 UTC m=+2738.962495746" watchObservedRunningTime="2026-11-02 20:34:17.996751165 +0000 UTC m=+2738.994439776"
Nov 02 20:34:18.021759 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:18.021661    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:18.115855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:18.110561    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:18.116321 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:18.116182    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:34:18.116321 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:18.116253    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 20:34:18.556797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:18.556343    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 20:34:18.556797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:18.556422    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79" podUID="e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 20:34:18.584789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:18.584263    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 20:34:18.584789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:18.584350    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9" podUID="23ca0a07-1d6a-49d1-9f58-aeef040acc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 20:34:19.083135 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.083075    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg" event={"ID":"3317681e-cd5b-4c6f-a8cb-a3dec259764e","Type":"ContainerStarted","Data":"5180b194254b4ed189c14c0624397b8592d791cd963486c6b603f16cf67e8ca5"}
Nov 02 20:34:19.113320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.113259    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:34:19.113560 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.113350    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 20:34:19.147066 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.146944    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:19.168307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.168269    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 02 20:34:19.168710 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.168666    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg" podUID="3317681e-cd5b-4c6f-a8cb-a3dec259764e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 02 20:34:19.234619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.234544    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg" podStartSLOduration=3.2345189579999998 podStartE2EDuration="3.234518958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:19.214067449 +0000 UTC m=+2740.211756057" watchObservedRunningTime="2026-11-02 20:34:19.234518958 +0000 UTC m=+2740.232207568"
Nov 02 20:34:19.533952 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.533912    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:19.562111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.562067    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 20:34:19.562442 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.562407    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79" podUID="e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 20:34:19.601353 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.601316    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 20:34:19.602020 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.601985    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9" podUID="23ca0a07-1d6a-49d1-9f58-aeef040acc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 20:34:19.792858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.789968    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6" podUID="f67432d1-22af-458b-bd57-002a02c62a5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:34:19.834793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.824905    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:34:19.871791 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:19.871341    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg" podUID="c2b4bca4-928d-439e-b582-979ff1768fc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:34:20.164863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.157896    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:34:20.165157 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.165116    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 20:34:20.167992 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.167941    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 02 20:34:20.168246 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.168199    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg" podUID="3317681e-cd5b-4c6f-a8cb-a3dec259764e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 02 20:34:20.198092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.197970    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:20.385353 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.385311    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:20.558796 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.557377    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 20:34:20.558796 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.557523    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79" podUID="e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 20:34:20.588796 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.588210    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 20:34:20.588796 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.588278    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9" podUID="23ca0a07-1d6a-49d1-9f58-aeef040acc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 20:34:20.619793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.618879    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"]
Nov 02 20:34:20.624226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.624185    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.682989 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.682018    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-bbee67-openshift-default-dockercfg-lw7rw\""
Nov 02 20:34:20.703803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.700493    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.703803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.700600    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.703803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.700641    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.703803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.700706    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.703803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.700742    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.703803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.700812    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.703803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.700867    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.703803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.700893    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.703803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.700925    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmn6p\" (UniqueName: \"kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-kube-api-access-wmn6p\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.718103 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.718064    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"]
Nov 02 20:34:20.822859 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.804705    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.856972 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.856934    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.870963 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.870920    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.899796 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.896243    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.923633 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.923593    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.924437 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.924410    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmn6p\" (UniqueName: \"kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-kube-api-access-wmn6p\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.925162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.925136    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.925397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.925376    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.939814 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.934024    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.939814 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.927151    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.939814 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.899223    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.939814 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.809288    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.939814 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.870650    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.939814 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.934580    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.939814 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.937596    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.940358 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.940131    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.941090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.941059    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:20.989794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:20.989665    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmn6p\" (UniqueName: \"kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-kube-api-access-wmn6p\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:21.110375 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:21.110232    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:34:21.110375 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:21.110302    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 20:34:21.131793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:21.127139    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 02 20:34:21.131793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:21.127221    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg" podUID="3317681e-cd5b-4c6f-a8cb-a3dec259764e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 02 20:34:21.239795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:21.239268    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:22.121090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.120247    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:34:22.121090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.120311    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 20:34:22.126292 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.126245    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 02 20:34:22.126598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.126557    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg" podUID="3317681e-cd5b-4c6f-a8cb-a3dec259764e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 02 20:34:22.140842 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.140810    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:22.142196 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.142163    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:22.142863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.142841    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:34:22.217413 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.216443    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:22.217642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.217503    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:22.218758 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.218723    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:34:22.325831 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:34:22.325797    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389ac8d7_6a0d_4007_a07d_d0851b28ab5e.slice/crio-46d10816198212ccd89fa446a26aa56fc9533f3e1282755a7e2fb755054df305 WatchSource:0}: Error finding container 46d10816198212ccd89fa446a26aa56fc9533f3e1282755a7e2fb755054df305: Status 404 returned error can't find the container with id 46d10816198212ccd89fa446a26aa56fc9533f3e1282755a7e2fb755054df305
Nov 02 20:34:22.328739 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.328705    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:22.328852 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.328792    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:22.328852 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.328824    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:34:22.343140 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.343111    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"]
Nov 02 20:34:22.796794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:22.795958    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:23.110447 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:23.110345    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:34:23.110447 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:23.110405    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 20:34:23.126652 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:23.126618    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 02 20:34:23.127169 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:23.127135    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg" podUID="3317681e-cd5b-4c6f-a8cb-a3dec259764e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 02 20:34:23.230593 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:23.230547    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq" event={"ID":"389ac8d7-6a0d-4007-a07d-d0851b28ab5e","Type":"ContainerStarted","Data":"bdbb882e99d107f61054f4cfa7e9aa42c5c69da135a49fedf3372f30f027e48b"}
Nov 02 20:34:23.231240 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:23.231204    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq" event={"ID":"389ac8d7-6a0d-4007-a07d-d0851b28ab5e","Type":"ContainerStarted","Data":"46d10816198212ccd89fa446a26aa56fc9533f3e1282755a7e2fb755054df305"}
Nov 02 20:34:23.240780 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:23.240729    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:23.241457 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:23.241433    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 20:34:23.241626 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:23.241609    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq" podUID="389ac8d7-6a0d-4007-a07d-d0851b28ab5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 20:34:23.365023 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:23.364876    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq" podStartSLOduration=3.364840508 podStartE2EDuration="3.364840508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:23.363026084 +0000 UTC m=+2744.360714697" watchObservedRunningTime="2026-11-02 20:34:23.364840508 +0000 UTC m=+2744.362529127"
Nov 02 20:34:23.843794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:23.841827    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:24.044789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:24.044570    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:24.121854 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:24.121394    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:34:24.121854 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:24.121469    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 20:34:24.142791 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:24.126332    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 02 20:34:24.142791 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:24.126431    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg" podUID="3317681e-cd5b-4c6f-a8cb-a3dec259764e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 02 20:34:24.255541 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:24.255504    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 20:34:24.260092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:24.260048    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq" podUID="389ac8d7-6a0d-4007-a07d-d0851b28ab5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 20:34:25.110139 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:25.110095    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:34:25.110394 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:25.110188    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 20:34:25.126974 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:25.126938    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 02 20:34:25.127217 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:25.126995    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg" podUID="3317681e-cd5b-4c6f-a8cb-a3dec259764e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 02 20:34:25.241019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:25.240972    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 20:34:25.241435 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:25.241044    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq" podUID="389ac8d7-6a0d-4007-a07d-d0851b28ab5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 20:34:25.907792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:25.905535    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:26.241157 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:26.241056    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 20:34:26.241157 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:26.241129    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq" podUID="389ac8d7-6a0d-4007-a07d-d0851b28ab5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 20:34:26.486501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:26.486442    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:26.487525 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:26.487495    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:26.488521 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:26.488495    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:34:27.159795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:27.155891    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:34:27.159795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:27.155967    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:34:27.240169 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:27.240134    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 20:34:27.240365 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:27.240192    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq" podUID="389ac8d7-6a0d-4007-a07d-d0851b28ab5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 20:34:27.403215 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:34:27.403154    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:34:27.677582 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:27.677537    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:27.745354 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:27.745316    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:27.745759 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:27.745733    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:27.746639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:27.746618    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:34:27.955558 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:27.955447    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:28.175786 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:28.175554    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:28.240489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:28.240389    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 20:34:28.240489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:28.240441    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq" podUID="389ac8d7-6a0d-4007-a07d-d0851b28ab5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 20:34:28.379632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:28.379558    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:28.570421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:28.570376    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:29.240304 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:29.240266    5224 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 20:34:29.240501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:29.240325    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq" podUID="389ac8d7-6a0d-4007-a07d-d0851b28ab5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 20:34:30.456514 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:30.456483    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:30.456973 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:30.456840    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:30.457726 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:30.457702    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:34:31.105926 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:31.105882    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:31.292321 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:31.292281    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:31.464261 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:31.464166    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:32.680694 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:32.680648    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:32.863752 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:32.863714    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:33.042756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:33.042717    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:36.837282 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:36.837238    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:37.031023 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:37.030980    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:38.267852 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:38.267815    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:40.088223 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.088179    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:40.265515 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.265471    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:40.596066 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.596026    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"]
Nov 02 20:34:40.600067 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.600045    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"
Nov 02 20:34:40.652276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.652243    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"]
Nov 02 20:34:40.653432 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.653389    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-a9ed60\""
Nov 02 20:34:40.700448 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.700417    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-a9ed60-dockercfg-vvrcp\""
Nov 02 20:34:40.712171 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.712141    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48c6q\" (UniqueName: \"kubernetes.io/projected/78e38acc-d4d7-43eb-abc8-323a9723f67e-kube-api-access-48c6q\") pod \"payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf\" (UID: \"78e38acc-d4d7-43eb-abc8-323a9723f67e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"
Nov 02 20:34:40.712349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.712192    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e38acc-d4d7-43eb-abc8-323a9723f67e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf\" (UID: \"78e38acc-d4d7-43eb-abc8-323a9723f67e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"
Nov 02 20:34:40.813629 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.813595    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48c6q\" (UniqueName: \"kubernetes.io/projected/78e38acc-d4d7-43eb-abc8-323a9723f67e-kube-api-access-48c6q\") pod \"payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf\" (UID: \"78e38acc-d4d7-43eb-abc8-323a9723f67e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"
Nov 02 20:34:40.813872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.813645    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e38acc-d4d7-43eb-abc8-323a9723f67e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf\" (UID: \"78e38acc-d4d7-43eb-abc8-323a9723f67e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"
Nov 02 20:34:40.814401 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.814379    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e38acc-d4d7-43eb-abc8-323a9723f67e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf\" (UID: \"78e38acc-d4d7-43eb-abc8-323a9723f67e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"
Nov 02 20:34:40.871696 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.871614    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48c6q\" (UniqueName: \"kubernetes.io/projected/78e38acc-d4d7-43eb-abc8-323a9723f67e-kube-api-access-48c6q\") pod \"payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf\" (UID: \"78e38acc-d4d7-43eb-abc8-323a9723f67e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"
Nov 02 20:34:40.912471 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:40.912435    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"
Nov 02 20:34:41.280694 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:34:41.278652    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e38acc_d4d7_43eb_abc8_323a9723f67e.slice/crio-d302a7e5d7d5fd9663dfdbdd48320cc2616aa1c8e7b43af6c669ec4ab5cdb7e9 WatchSource:0}: Error finding container d302a7e5d7d5fd9663dfdbdd48320cc2616aa1c8e7b43af6c669ec4ab5cdb7e9: Status 404 returned error can't find the container with id d302a7e5d7d5fd9663dfdbdd48320cc2616aa1c8e7b43af6c669ec4ab5cdb7e9
Nov 02 20:34:41.287001 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:41.286976    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"]
Nov 02 20:34:41.404522 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:41.404484    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf" event={"ID":"78e38acc-d4d7-43eb-abc8-323a9723f67e","Type":"ContainerStarted","Data":"d7efb59b0d091a5f57b52b04ff5a41cc090b07f39047913e29335ded2308199e"}
Nov 02 20:34:41.404748 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:41.404546    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf" event={"ID":"78e38acc-d4d7-43eb-abc8-323a9723f67e","Type":"ContainerStarted","Data":"d302a7e5d7d5fd9663dfdbdd48320cc2616aa1c8e7b43af6c669ec4ab5cdb7e9"}
Nov 02 20:34:41.404748 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:41.404576    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"
Nov 02 20:34:41.521558 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:41.521494    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf" podStartSLOduration=1.521475388 podStartE2EDuration="1.521475388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:41.518813165 +0000 UTC m=+2762.516501773" watchObservedRunningTime="2026-11-02 20:34:41.521475388 +0000 UTC m=+2762.519163998"
Nov 02 20:34:42.063181 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:42.063142    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:47.044491 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:47.044450    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:51.657832 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.657791    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"]
Nov 02 20:34:51.661462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.661443    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:51.714389 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.714350    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"]
Nov 02 20:34:51.751108 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.751066    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:51.751313 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.751232    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:51.751399 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.751376    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjkw7\" (UniqueName: \"kubernetes.io/projected/7689b623-5d86-473a-82aa-0739d899ff06-kube-api-access-wjkw7\") pod \"maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:51.751465 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.751449    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7689b623-5d86-473a-82aa-0739d899ff06-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:51.762692 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.762661    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-44a484-serving-cert\""
Nov 02 20:34:51.762883 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.762731    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 20:34:51.762883 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.762763    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 20:34:51.766060 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.766043    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 20:34:51.766301 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.766285    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nltm4\""
Nov 02 20:34:51.853040 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.853001    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:51.853248 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.853108    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjkw7\" (UniqueName: \"kubernetes.io/projected/7689b623-5d86-473a-82aa-0739d899ff06-kube-api-access-wjkw7\") pod \"maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:51.853248 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.853151    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7689b623-5d86-473a-82aa-0739d899ff06-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:51.853248 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.853212    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:51.853404 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:34:51.853348    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-serving-cert: secret "maas-api-e2e-shared-a-w1-44a484-serving-cert" not found
Nov 02 20:34:51.853453 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:34:51.853428    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-maas-api-tls podName:7689b623-5d86-473a-82aa-0739d899ff06 nodeName:}" failed. No retries permitted until 2026-11-02 20:34:52.353403854 +0000 UTC m=+2773.351092445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-maas-api-tls") pod "maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c" (UID: "7689b623-5d86-473a-82aa-0739d899ff06") : secret "maas-api-e2e-shared-a-w1-44a484-serving-cert" not found
Nov 02 20:34:51.853904 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.853880    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7689b623-5d86-473a-82aa-0739d899ff06-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:51.855545 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.855513    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:51.897987 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.897945    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:51.923395 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:51.920990    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjkw7\" (UniqueName: \"kubernetes.io/projected/7689b623-5d86-473a-82aa-0739d899ff06-kube-api-access-wjkw7\") pod \"maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:52.082348 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:52.082310    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:52.250163 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:52.250066    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:52.360930 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:52.360882    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:52.363454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:52.363423    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:52.413935 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:52.413906    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"
Nov 02 20:34:52.574927 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:52.574885    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:52.950279 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:34:52.950245    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7689b623_5d86_473a_82aa_0739d899ff06.slice/crio-43c51ba1fb19fc45732491fdf7f4086ebc650353081cc8af2c6dd3307a3d5aab WatchSource:0}: Error finding container 43c51ba1fb19fc45732491fdf7f4086ebc650353081cc8af2c6dd3307a3d5aab: Status 404 returned error can't find the container with id 43c51ba1fb19fc45732491fdf7f4086ebc650353081cc8af2c6dd3307a3d5aab
Nov 02 20:34:52.951823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:52.951799    5224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:34:52.953647 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:52.953621    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"]
Nov 02 20:34:53.478193 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:53.478155    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c" event={"ID":"7689b623-5d86-473a-82aa-0739d899ff06","Type":"ContainerStarted","Data":"43c51ba1fb19fc45732491fdf7f4086ebc650353081cc8af2c6dd3307a3d5aab"}
Nov 02 20:34:54.486733 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:54.486693    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c" event={"ID":"7689b623-5d86-473a-82aa-0739d899ff06","Type":"ContainerStarted","Data":"73860c8070484b0a9d8a90b3ddb3dbd46008e800a221797368d6f716035638e0"}
Nov 02 20:34:54.487220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:54.486908    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:34:54.604724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:54.604661    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c" podStartSLOduration=2.724582454 podStartE2EDuration="3.604644905s" totalImagesPullingTime="880.062451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:54.602142604 +0000 UTC m=+2775.599831215" watchObservedRunningTime="2026-11-02 20:34:54.604644905 +0000 UTC m=+2775.602333524"
Nov 02 20:34:57.011708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:34:57.011637    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:34:57.360628 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:34:57.360589    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:35:00.305983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:00.305939    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:00.499613 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:00.499584    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 20:35:01.898019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:01.897978    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:06.777360 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:06.777321    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:07.017924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:07.017883    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:35:11.685276 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:11.685234    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:16.632138 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:16.632046    5224 generic.go:356] "Generic (PLEG): container finished" podID="4b4880ac-7310-4d51-86f2-75634f425b28" containerID="ae092d241cf7cfcbe00d35b7886d7400707e30c0f02e2202cb31ae47ad91076e" exitCode=0
Nov 02 20:35:16.632138 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:16.632122    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh" event={"ID":"4b4880ac-7310-4d51-86f2-75634f425b28","Type":"ContainerDied","Data":"ae092d241cf7cfcbe00d35b7886d7400707e30c0f02e2202cb31ae47ad91076e"}
Nov 02 20:35:16.632611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:16.632591    5224 scope.go:122] "RemoveContainer" containerID="ae092d241cf7cfcbe00d35b7886d7400707e30c0f02e2202cb31ae47ad91076e"
Nov 02 20:35:16.634247 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:16.634217    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:35:16.634317 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:16.634290    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:35:16.634353 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:16.634337    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:35:16.966849 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:16.966684    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:17.568008 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:17.567892    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:35:17.640571 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:17.640529    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh" event={"ID":"4b4880ac-7310-4d51-86f2-75634f425b28","Type":"ContainerStarted","Data":"5b84ea2125a171259fa0a6b7655de97f40b9e7adddb43d23c0b1801260dd6630"}
Nov 02 20:35:18.375323 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:18.375222    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:35:18.376420 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:18.376390    5224 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 20:35:18.376612 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:18.376442    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh" podUID="4b4880ac-7310-4d51-86f2-75634f425b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 20:35:19.374972 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:19.374929    5224 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 20:35:19.375435 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:19.374993    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh" podUID="4b4880ac-7310-4d51-86f2-75634f425b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 20:35:20.374779 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:20.374738    5224 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 20:35:20.374999 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:20.374875    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh" podUID="4b4880ac-7310-4d51-86f2-75634f425b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 20:35:21.375373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:21.375330    5224 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 20:35:21.376066 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:21.376026    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh" podUID="4b4880ac-7310-4d51-86f2-75634f425b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 20:35:21.818319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:21.818273    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:21.989002 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:21.988936    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:22.304719 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:22.304677    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:22.375350 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:22.375298    5224 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 20:35:22.375586 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:22.375375    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh" podUID="4b4880ac-7310-4d51-86f2-75634f425b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 20:35:23.380641 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:23.380608    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:35:23.381110 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:23.381031    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:35:23.381736 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:23.381715    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 20:35:27.148593 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:27.148551    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:27.359987 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:35:27.359951    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:35:30.249153 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:30.249116    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:31.739944 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.739902    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"]
Nov 02 20:35:31.742600 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.742579    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:31.796611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.796576    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"]
Nov 02 20:35:31.844970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.844932    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-a9ed60-serving-cert\""
Nov 02 20:35:31.866615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.866579    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb0be65-a5fd-4ea9-8573-d9a755151a44-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:31.866820 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.866761    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:31.866875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.866835    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72nw7\" (UniqueName: \"kubernetes.io/projected/bdb0be65-a5fd-4ea9-8573-d9a755151a44-kube-api-access-72nw7\") pod \"maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:31.866937 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.866916    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:31.968002 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.967953    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:31.968239 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.968045    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb0be65-a5fd-4ea9-8573-d9a755151a44-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:31.968239 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:35:31.968118    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-serving-cert: secret "maas-api-e2e-shared-a-w4-a9ed60-serving-cert" not found
Nov 02 20:35:31.968239 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.968155    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:31.968239 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.968178    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72nw7\" (UniqueName: \"kubernetes.io/projected/bdb0be65-a5fd-4ea9-8573-d9a755151a44-kube-api-access-72nw7\") pod \"maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:31.968239 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:35:31.968201    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-maas-api-tls podName:bdb0be65-a5fd-4ea9-8573-d9a755151a44 nodeName:}" failed. No retries permitted until 2026-11-02 20:35:32.468182336 +0000 UTC m=+2813.465870923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-maas-api-tls") pod "maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj" (UID: "bdb0be65-a5fd-4ea9-8573-d9a755151a44") : secret "maas-api-e2e-shared-a-w4-a9ed60-serving-cert" not found
Nov 02 20:35:31.968875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.968837    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb0be65-a5fd-4ea9-8573-d9a755151a44-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:31.970816 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.970790    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:31.972301 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:31.972266    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:32.026661 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:32.026624    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72nw7\" (UniqueName: \"kubernetes.io/projected/bdb0be65-a5fd-4ea9-8573-d9a755151a44-kube-api-access-72nw7\") pod \"maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:32.188559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:32.188517    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:32.361614 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:32.361505    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:32.475393 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:32.475333    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:32.478015 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:32.477989    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:32.655520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:32.655414    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:33.031588 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:35:33.031555    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb0be65_a5fd_4ea9_8573_d9a755151a44.slice/crio-0f89d36b3934a9acef577fecd7e58356ffe57f6af990888b413e3c8e1f8917d4 WatchSource:0}: Error finding container 0f89d36b3934a9acef577fecd7e58356ffe57f6af990888b413e3c8e1f8917d4: Status 404 returned error can't find the container with id 0f89d36b3934a9acef577fecd7e58356ffe57f6af990888b413e3c8e1f8917d4
Nov 02 20:35:33.034354 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:33.034322    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"]
Nov 02 20:35:33.773073 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:33.773035    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj" event={"ID":"bdb0be65-a5fd-4ea9-8573-d9a755151a44","Type":"ContainerStarted","Data":"0f89d36b3934a9acef577fecd7e58356ffe57f6af990888b413e3c8e1f8917d4"}
Nov 02 20:35:34.434660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:34.434613    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:35:34.781343 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:34.781303    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj" event={"ID":"bdb0be65-a5fd-4ea9-8573-d9a755151a44","Type":"ContainerStarted","Data":"fa47e8e2863cb6ee27e8b40164653a251c91d317f985ab8002ba971884172c30"}
Nov 02 20:35:34.781556 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:34.781394    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:34.900851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:34.900795    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj" podStartSLOduration=3.079810408 podStartE2EDuration="3.900780688s" totalImagesPullingTime="820.97028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:34.899568659 +0000 UTC m=+2815.897257267" watchObservedRunningTime="2026-11-02 20:35:34.900780688 +0000 UTC m=+2815.898469290"
Nov 02 20:35:37.146118 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:37.146072    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:40.794257 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:40.794224    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 20:35:42.150528 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:42.150468    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:46.514387 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:46.514331    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:46.923757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:46.923666    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:47.253669 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:47.253566    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:47.540204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:47.540163    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:47.999592 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:47.999481    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:51.709719 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:51.709665    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:51.942558 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:51.942517    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:53.388755 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:53.388703    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:53.872819 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:53.872748    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:54.870411 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:54.870361    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:55.047197 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:55.047136    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:35:57.368688 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:35:57.368647    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:35:57.846890 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:35:57.846851    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:00.597064 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:00.597023    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:01.377280 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:01.377235    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:01.912603 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:01.912557    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:02.375069 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:02.375026    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:02.862891 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:02.862856    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:36:03.083678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.083636    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:03.373640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.373599    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"]
Nov 02 20:36:03.376993 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.376966    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.433067 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.430807    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-09ecd0-openshift-default-dockercfg-ccwlp\""
Nov 02 20:36:03.465429 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.465384    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fd9fb0-acae-4989-be05-0c80707435ed-istiod-ca-cert\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.465644 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.465456    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-data\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.465722 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.465667    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-certs\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.465722 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.465696    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fd9fb0-acae-4989-be05-0c80707435ed-istio-podinfo\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.465874 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.465762    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq4bw\" (UniqueName: \"kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-kube-api-access-sq4bw\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.465874 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.465845    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-envoy\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.465976 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.465875    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.465976 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.465899    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-credential-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.465976 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.465960    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-istio-token\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.484714 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.484668    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"]
Nov 02 20:36:03.508123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.508076    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:03.569789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.569445    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-envoy\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.569789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.569509    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.569789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.569534    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-credential-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.569789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.569588    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-istio-token\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.569789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.569656    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fd9fb0-acae-4989-be05-0c80707435ed-istiod-ca-cert\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.569789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.569715    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-data\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.570280 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.569881    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-certs\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.570280 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.569905    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fd9fb0-acae-4989-be05-0c80707435ed-istio-podinfo\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.570280 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.569929    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq4bw\" (UniqueName: \"kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-kube-api-access-sq4bw\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.571293 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.570555    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-certs\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.571293 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.570858    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-data\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.572222 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.572198    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.577221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.577187    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-envoy\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.581197 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.581164    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-credential-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.590319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.590277    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fd9fb0-acae-4989-be05-0c80707435ed-istio-podinfo\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.593543 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.593503    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fd9fb0-acae-4989-be05-0c80707435ed-istiod-ca-cert\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.628741 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.628656    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq4bw\" (UniqueName: \"kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-kube-api-access-sq4bw\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.628980 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.628751    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-istio-token\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.690778 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.690720    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:03.766442 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:03.766401    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:04.130147 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:36:04.130111    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11fd9fb0_acae_4989_be05_0c80707435ed.slice/crio-a951dc096ef208fd164423d775429dd0337f3685f73f37b3e069531b4ee17954 WatchSource:0}: Error finding container a951dc096ef208fd164423d775429dd0337f3685f73f37b3e069531b4ee17954: Status 404 returned error can't find the container with id a951dc096ef208fd164423d775429dd0337f3685f73f37b3e069531b4ee17954
Nov 02 20:36:04.132870 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:04.132835    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:36:04.132976 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:04.132920    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:36:04.132976 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:04.132951    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:36:04.133314 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:04.133283    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"]
Nov 02 20:36:04.979101 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:04.979059    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" event={"ID":"11fd9fb0-acae-4989-be05-0c80707435ed","Type":"ContainerStarted","Data":"5076ceea02f90b6461c786511cc1fe4b576f9ebc42e919ac8d2d81aa3a16ab22"}
Nov 02 20:36:04.979312 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:04.979131    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" event={"ID":"11fd9fb0-acae-4989-be05-0c80707435ed","Type":"ContainerStarted","Data":"a951dc096ef208fd164423d775429dd0337f3685f73f37b3e069531b4ee17954"}
Nov 02 20:36:05.097597 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:05.097539    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" podStartSLOduration=2.09752445 podStartE2EDuration="2.09752445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:05.096412985 +0000 UTC m=+2846.094101593" watchObservedRunningTime="2026-11-02 20:36:05.09752445 +0000 UTC m=+2846.095213059"
Nov 02 20:36:05.691671 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:05.691637    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:05.697935 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:05.697904    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:05.985405 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:05.985304    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:05.986434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:05.986415    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:36:08.370486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:08.370442    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:08.553087 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:08.553050    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:08.756982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:08.756888    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:11.610329 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:11.610283    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:11.785619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:11.785567    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:14.013295 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:14.013241    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:19.095657 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:19.095612    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:23.953910 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:23.953870    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:27.360825 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:36:27.360783    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:36:28.858907 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:28.858865    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:30.094452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:30.094414    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:30.524857 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:30.524809    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:31.085146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:31.085095    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:31.313489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:31.313443    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:33.995536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:33.995491    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:34.164901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:34.164864    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:36:35.650134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:35.650094    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:35.842700 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:35.842658    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:36.015943 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:36.015908    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:38.852173 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:38.852132    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:39.239243 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:39.239129    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:40.373624 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:40.373584    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:40.802993 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:40.802952    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:41.444521 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:41.444478    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:41.938695 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:41.938654    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:43.888432 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:43.888391    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:44.786422 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:44.786375    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:46.284850 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:46.284808    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:46.477173 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:46.477126    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:46.661126 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:46.661021    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:50.074387 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:50.074288    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:50.992379 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:50.992338    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:51.208989 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:51.208950    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:54.863119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:54.863077    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"]
Nov 02 20:36:54.866214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:54.866194    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:54.919562 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:54.919521    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"]
Nov 02 20:36:54.955959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:54.955918    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375bdcc1-e847-4275-85b8-6a0197b5cb66-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:54.956164 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:54.956074    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmw5\" (UniqueName: \"kubernetes.io/projected/375bdcc1-e847-4275-85b8-6a0197b5cb66-kube-api-access-6bmw5\") pod \"maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:54.956219 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:54.956171    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:54.956219 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:54.956198    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:54.967879 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:54.967848    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-78bef7-serving-cert\""
Nov 02 20:36:55.057501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:55.057466    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmw5\" (UniqueName: \"kubernetes.io/projected/375bdcc1-e847-4275-85b8-6a0197b5cb66-kube-api-access-6bmw5\") pod \"maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:55.057703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:55.057537    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:55.057703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:55.057561    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:55.057703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:55.057631    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375bdcc1-e847-4275-85b8-6a0197b5cb66-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:55.057703 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:36:55.057682    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-serving-cert: secret "maas-api-e2e-shared-b-w2-78bef7-serving-cert" not found
Nov 02 20:36:55.058393 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:36:55.057796    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-maas-api-tls podName:375bdcc1-e847-4275-85b8-6a0197b5cb66 nodeName:}" failed. No retries permitted until 2026-11-02 20:36:55.557747343 +0000 UTC m=+2896.555435929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-maas-api-tls") pod "maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl" (UID: "375bdcc1-e847-4275-85b8-6a0197b5cb66") : secret "maas-api-e2e-shared-b-w2-78bef7-serving-cert" not found
Nov 02 20:36:55.058393 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:55.058287    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375bdcc1-e847-4275-85b8-6a0197b5cb66-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:55.060159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:55.060139    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:55.104619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:55.104569    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:55.119234 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:55.119142    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmw5\" (UniqueName: \"kubernetes.io/projected/375bdcc1-e847-4275-85b8-6a0197b5cb66-kube-api-access-6bmw5\") pod \"maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:55.324059 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:55.324020    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:36:55.564128 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:55.564080    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:55.566903 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:55.566865    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:55.781520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:55.781479    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:36:56.156724 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:36:56.156688    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375bdcc1_e847_4275_85b8_6a0197b5cb66.slice/crio-444f7fa31db53538f4d9d19ffa0834a0a981d599428fbd1e5e514acb62fabd6c WatchSource:0}: Error finding container 444f7fa31db53538f4d9d19ffa0834a0a981d599428fbd1e5e514acb62fabd6c: Status 404 returned error can't find the container with id 444f7fa31db53538f4d9d19ffa0834a0a981d599428fbd1e5e514acb62fabd6c
Nov 02 20:36:56.160177 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:56.160151    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"]
Nov 02 20:36:56.322600 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:36:56.322560    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl" event={"ID":"375bdcc1-e847-4275-85b8-6a0197b5cb66","Type":"ContainerStarted","Data":"444f7fa31db53538f4d9d19ffa0834a0a981d599428fbd1e5e514acb62fabd6c"}
Nov 02 20:36:57.140818 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:36:57.140754    5224 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image: copying system image from manifest list: parsing image configuration: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/67/673872e90110ca32b242313220b9d52506c1e646e315d8ea3b6239c0d5c7db06?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261102%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261102T203656Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=f2dcef0169a5a4ec672b7dc43356c0b7c7bbaf1714465f312e30a843d5369230&region=us-east-1&namespace=opendatahub&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=maas-api&akamai_signature=exp=1793652716~hmac=c534c8fd73700980e6208be02dd67e2e4ec3d675d1280221ae10b0976be27018\": read tcp 10.0.134.196:54320->23.46.228.109:443: read: connection reset by peer" image="quay.io/opendatahub/maas-api@sha256:38856cc4c7a177adc093011b162369e0356a72e1b5e1aeed7aff85b1a3d3b07b"
Nov 02 20:36:57.141038 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:36:57.141009    5224 kuberuntime_manager.go:1707] "Unhandled Error" err="container maas-api start failed in pod maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl_odh-ai-gateway-infra(375bdcc1-e847-4275-85b8-6a0197b5cb66): ErrImagePull: unable to pull image: copying system image from manifest list: parsing image configuration: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/67/673872e90110ca32b242313220b9d52506c1e646e315d8ea3b6239c0d5c7db06?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261102%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261102T203656Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=f2dcef0169a5a4ec672b7dc43356c0b7c7bbaf1714465f312e30a843d5369230&region=us-east-1&namespace=opendatahub&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=maas-api&akamai_signature=exp=1793652716~hmac=c534c8fd73700980e6208be02dd67e2e4ec3d675d1280221ae10b0976be27018\": read tcp 10.0.134.196:54320->23.46.228.109:443: read: connection reset by peer" logger="UnhandledError"
Nov 02 20:36:57.141101 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:36:57.141067    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with ErrImagePull: \"unable to pull image: copying system image from manifest list: parsing image configuration: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/67/673872e90110ca32b242313220b9d52506c1e646e315d8ea3b6239c0d5c7db06?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261102%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261102T203656Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=f2dcef0169a5a4ec672b7dc43356c0b7c7bbaf1714465f312e30a843d5369230&region=us-east-1&namespace=opendatahub&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=maas-api&akamai_signature=exp=1793652716~hmac=c534c8fd73700980e6208be02dd67e2e4ec3d675d1280221ae10b0976be27018\\\": read tcp 10.0.134.196:54320->23.46.228.109:443: read: connection reset by peer\"" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl" podUID="375bdcc1-e847-4275-85b8-6a0197b5cb66"
Nov 02 20:36:57.329355 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:36:57.329309    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/opendatahub/maas-api@sha256:38856cc4c7a177adc093011b162369e0356a72e1b5e1aeed7aff85b1a3d3b07b\\\": ErrImagePull: unable to pull image: copying system image from manifest list: parsing image configuration: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/67/673872e90110ca32b242313220b9d52506c1e646e315d8ea3b6239c0d5c7db06?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261102%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261102T203656Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=f2dcef0169a5a4ec672b7dc43356c0b7c7bbaf1714465f312e30a843d5369230&region=us-east-1&namespace=opendatahub&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=maas-api&akamai_signature=exp=1793652716~hmac=c534c8fd73700980e6208be02dd67e2e4ec3d675d1280221ae10b0976be27018\\\": read tcp 10.0.134.196:54320->23.46.228.109:443: read: connection reset by peer\"" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl" podUID="375bdcc1-e847-4275-85b8-6a0197b5cb66"
Nov 02 20:36:57.359835 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:36:57.359800    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:37:00.155979 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:00.155931    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:00.935827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:00.935758    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:01.707500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:01.707456    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:05.048108 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:05.048058    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:09.888658 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:09.888618    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:11.414826 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:11.414715    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl" event={"ID":"375bdcc1-e847-4275-85b8-6a0197b5cb66","Type":"ContainerStarted","Data":"2f3e3a836d08704d91d34ec451c12f91cc766dc5fd0c2248cd0a520c3ccb4ea9"}
Nov 02 20:37:11.415302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:11.414946    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:37:11.535430 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:11.535367    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl" podStartSLOduration=2.675501707 podStartE2EDuration="17.535350863s" totalImagesPullingTime="14.859849156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-11-02 20:37:11.53262327 +0000 UTC m=+2912.530311878" watchObservedRunningTime="2026-11-02 20:37:11.535350863 +0000 UTC m=+2912.533039472"
Nov 02 20:37:14.598037 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:14.597992    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:14.770673 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:14.770633    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:14.943618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:14.943516    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:17.439085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:17.439051    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 20:37:19.602982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:19.602926    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:19.888646 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:19.888543    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:37:20.685804 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:20.684833    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:21.208076 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:21.208028    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:21.557282 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:21.557237    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:21.741933 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:21.741887    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:25.000737 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:25.000688    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:25.314615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:25.314576    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:25.486262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:25.486226    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:26.717721 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:26.717672    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:27.359840 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:37:27.359804    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:37:29.886397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:29.886351    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:32.149471 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:32.149428    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:34.748824 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:34.748756    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:39.577659 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:39.577614    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:44.374716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:44.374675    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:44.600369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:44.600331    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:49.269873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.269758    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"]
Nov 02 20:37:49.277729 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.277706    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.369119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.369055    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-podinfo\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.370642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.369150    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-token\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.370642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.369251    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k92f8\" (UniqueName: \"kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-kube-api-access-k92f8\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.370642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.369288    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-credential-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.370642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.369329    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-envoy\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.370642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.369428    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.370642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.369551    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-certs\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.370642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.369587    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-data\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.370642 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.369647    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eab4cf3-570b-4827-b25d-c01a5683a048-istiod-ca-cert\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.422486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.422440    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:49.472484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.472442    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-token\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.472651 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.472500    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k92f8\" (UniqueName: \"kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-kube-api-access-k92f8\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.472651 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.472532    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-credential-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.472651 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.472572    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-envoy\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.472651 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.472629    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.472918 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.472784    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-certs\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.472918 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.472826    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-data\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.472918 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.472887    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eab4cf3-570b-4827-b25d-c01a5683a048-istiod-ca-cert\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.473083 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.472921    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"]
Nov 02 20:37:49.473083 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.472970    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-podinfo\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.474959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.473972    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-credential-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.474959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.474094    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-certs\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.474959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.474253    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-data\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.474959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.474353    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.474959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.474912    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eab4cf3-570b-4827-b25d-c01a5683a048-istiod-ca-cert\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.476929 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.476142    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-podinfo\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.476929 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.476378    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-envoy\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.531229 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.531156    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-token\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.531374 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.531257    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k92f8\" (UniqueName: \"kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-kube-api-access-k92f8\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.596934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.596836    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:49.666521 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.665161    5224 generic.go:356] "Generic (PLEG): container finished" podID="11fd9fb0-acae-4989-be05-0c80707435ed" containerID="5076ceea02f90b6461c786511cc1fe4b576f9ebc42e919ac8d2d81aa3a16ab22" exitCode=0
Nov 02 20:37:49.666521 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.665238    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" event={"ID":"11fd9fb0-acae-4989-be05-0c80707435ed","Type":"ContainerDied","Data":"5076ceea02f90b6461c786511cc1fe4b576f9ebc42e919ac8d2d81aa3a16ab22"}
Nov 02 20:37:49.666521 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.665723    5224 scope.go:122] "RemoveContainer" containerID="5076ceea02f90b6461c786511cc1fe4b576f9ebc42e919ac8d2d81aa3a16ab22"
Nov 02 20:37:49.669444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.669409    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:37:49.669572 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.669505    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:37:49.669572 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:49.669555    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:37:50.017967 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:37:50.017930    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eab4cf3_570b_4827_b25d_c01a5683a048.slice/crio-ca04bc95c46b7fbca715653bcb98dea98ff7e87e45fd86c7b9ed9c373330c6a0 WatchSource:0}: Error finding container ca04bc95c46b7fbca715653bcb98dea98ff7e87e45fd86c7b9ed9c373330c6a0: Status 404 returned error can't find the container with id ca04bc95c46b7fbca715653bcb98dea98ff7e87e45fd86c7b9ed9c373330c6a0
Nov 02 20:37:50.021358 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:50.021319    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:37:50.021506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:50.021410    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"]
Nov 02 20:37:50.021506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:50.021417    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:37:50.021583 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:50.021531    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:37:50.706452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:50.706391    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" event={"ID":"11fd9fb0-acae-4989-be05-0c80707435ed","Type":"ContainerStarted","Data":"0dda4596c0cc14bb5766d11d6a3cafae5749a02898b44f30e558feeda315c0a9"}
Nov 02 20:37:50.708823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:50.708760    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr" event={"ID":"0eab4cf3-570b-4827-b25d-c01a5683a048","Type":"ContainerStarted","Data":"7d80df9b2ec74d6f25a367c6b48dd84a284c5acc7bbc90286fb9c7ef32d3b0c5"}
Nov 02 20:37:50.708968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:50.708848    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr" event={"ID":"0eab4cf3-570b-4827-b25d-c01a5683a048","Type":"ContainerStarted","Data":"ca04bc95c46b7fbca715653bcb98dea98ff7e87e45fd86c7b9ed9c373330c6a0"}
Nov 02 20:37:50.946361 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:50.946297    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr" podStartSLOduration=1.946280785 podStartE2EDuration="1.946280785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:50.94481068 +0000 UTC m=+2951.942499292" watchObservedRunningTime="2026-11-02 20:37:50.946280785 +0000 UTC m=+2951.943969393"
Nov 02 20:37:50.987876 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:50.986891    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:37:51.598796 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:51.597952    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:51.599845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:51.599818    5224 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 02 20:37:51.599927 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:51.599882    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr" podUID="0eab4cf3-570b-4827-b25d-c01a5683a048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 02 20:37:51.695792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:51.695206    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:37:51.695792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:51.695355    5224 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 20:37:51.695792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:51.695405    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 20:37:52.599862 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:52.599474    5224 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 02 20:37:52.599862 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:52.599554    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr" podUID="0eab4cf3-570b-4827-b25d-c01a5683a048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 02 20:37:52.691632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:52.691592    5224 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 20:37:52.691987 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:52.691953    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 20:37:53.598004 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:53.597917    5224 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 02 20:37:53.598004 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:53.597993    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr" podUID="0eab4cf3-570b-4827-b25d-c01a5683a048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 02 20:37:53.694020 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:53.693891    5224 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 20:37:53.694020 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:53.693957    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 20:37:54.445514 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:54.444589    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:54.597795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:54.597287    5224 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 02 20:37:54.597795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:54.597363    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr" podUID="0eab4cf3-570b-4827-b25d-c01a5683a048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 02 20:37:54.694793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:54.694342    5224 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 20:37:54.694793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:54.694429    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 20:37:55.598412 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:55.598373    5224 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 02 20:37:55.598716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:55.598684    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr" podUID="0eab4cf3-570b-4827-b25d-c01a5683a048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 02 20:37:55.691803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:55.691300    5224 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 20:37:55.691803 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:55.691386    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 20:37:56.602677 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:56.602641    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:56.603171 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:56.603024    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:56.603737 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:56.603716    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 20:37:56.696923 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:56.696890    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:37:56.697942 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:56.697918    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:37:57.361686 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:37:57.361647    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:37:59.058378 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.058344    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"]
Nov 02 20:37:59.062033 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.062013    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.115542 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.115504    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"]
Nov 02 20:37:59.153168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.153131    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93a40bb-5ce3-4355-b2d4-6228d7857424-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.153397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.153192    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztswm\" (UniqueName: \"kubernetes.io/projected/f93a40bb-5ce3-4355-b2d4-6228d7857424-kube-api-access-ztswm\") pod \"maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.153397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.153327    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.153535 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.153456    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.174240 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.174201    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-578636-serving-cert\""
Nov 02 20:37:59.254312 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.254276    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.254532 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.254367    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.254532 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:37:59.254422    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-serving-cert: secret "maas-api-e2e-shared-b-w4-578636-serving-cert" not found
Nov 02 20:37:59.254532 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.254481    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93a40bb-5ce3-4355-b2d4-6228d7857424-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.254532 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:37:59.254507    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-maas-api-tls podName:f93a40bb-5ce3-4355-b2d4-6228d7857424 nodeName:}" failed. No retries permitted until 2026-11-02 20:37:59.75448502 +0000 UTC m=+2960.752173626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-maas-api-tls") pod "maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj" (UID: "f93a40bb-5ce3-4355-b2d4-6228d7857424") : secret "maas-api-e2e-shared-b-w4-578636-serving-cert" not found
Nov 02 20:37:59.254796 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.254644    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztswm\" (UniqueName: \"kubernetes.io/projected/f93a40bb-5ce3-4355-b2d4-6228d7857424-kube-api-access-ztswm\") pod \"maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.255230 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.255208    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93a40bb-5ce3-4355-b2d4-6228d7857424-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.257017 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.256986    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.315021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.314942    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztswm\" (UniqueName: \"kubernetes.io/projected/f93a40bb-5ce3-4355-b2d4-6228d7857424-kube-api-access-ztswm\") pod \"maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.315762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.315729    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:59.501177 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.501138    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:59.670860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.670738    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:37:59.760159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.760122    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.762760 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.762725    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:37:59.950300 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.950213    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:37:59.975844 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:37:59.975805    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:38:00.345036 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:38:00.345000    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93a40bb_5ce3_4355_b2d4_6228d7857424.slice/crio-e2c1afb961756e83d36dfce93db690353a6b62fda4a5640b92710cf1c8f878bf WatchSource:0}: Error finding container e2c1afb961756e83d36dfce93db690353a6b62fda4a5640b92710cf1c8f878bf: Status 404 returned error can't find the container with id e2c1afb961756e83d36dfce93db690353a6b62fda4a5640b92710cf1c8f878bf
Nov 02 20:38:00.348998 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:00.348961    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"]
Nov 02 20:38:00.772966 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:00.772928    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj" event={"ID":"f93a40bb-5ce3-4355-b2d4-6228d7857424","Type":"ContainerStarted","Data":"e2c1afb961756e83d36dfce93db690353a6b62fda4a5640b92710cf1c8f878bf"}
Nov 02 20:38:01.283074 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:01.283038    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:01.780553 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:01.780512    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj" event={"ID":"f93a40bb-5ce3-4355-b2d4-6228d7857424","Type":"ContainerStarted","Data":"c955b57dd6ced0ab67ab9a98bf3064a717c825e0ba4b5bd5dfdd4f89d90595d1"}
Nov 02 20:38:01.781003 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:01.780584    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:38:01.897876 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:01.897819    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj" podStartSLOduration=2.01196497 podStartE2EDuration="2.897802945s" totalImagesPullingTime="885.837975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:01.896036033 +0000 UTC m=+2962.893724642" watchObservedRunningTime="2026-11-02 20:38:01.897802945 +0000 UTC m=+2962.895491553"
Nov 02 20:38:02.303106 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:02.303056    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:04.252617 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:04.252576    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:05.674182 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:05.674126    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:07.792839 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:07.792806    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 20:38:10.322977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:10.322929    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:10.715588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:10.715486    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:10.892967 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:10.892926    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:11.503025 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:11.502989    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:12.773479 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:12.773435    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:12.995939 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:12.995901    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:16.207159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:16.207111    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:17.412849 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:17.412811    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:18.094666 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:18.094575    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:18.263495 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:18.263459    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:19.551394 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:19.551336    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:20.809022 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:20.808981    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:22.843479 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:22.843438    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:23.041298 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:23.041255    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:23.215679 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:23.215567    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:23.420064 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:23.420012    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:24.563416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:24.563380    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:25.019736 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:25.019696    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:27.358432 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:38:27.358399    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:38:28.120358 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:28.120317    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:32.469436 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:32.469401    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:38:33.375178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:33.372900    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:36.164216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:36.164181    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:38.562271 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:38.562234    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:43.571353 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:43.571310    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:47.053373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:47.053337    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"]
Nov 02 20:38:47.104539 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:47.104506    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ea1fe76-tmvp5"]
Nov 02 20:38:47.123972 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:47.123941    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:47.167633 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:47.167600    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d773d4-a9a5-43ca-b88f-8b0e3c303e2a" path="/var/lib/kubelet/pods/85d773d4-a9a5-43ca-b88f-8b0e3c303e2a/volumes"
Nov 02 20:38:47.984875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:47.984731    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:48.214434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:48.214397    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:48.437636 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:48.437591    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:51.704503 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:51.704460    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:51.890821 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:51.890750    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:52.073035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:52.072999    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:53.650009 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:53.649875    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:54.893283 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:54.893237    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:55.071335 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:55.071300    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:38:57.360199 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:38:57.360152    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:38:58.984483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:38:58.984440    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:01.392110 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:01.392075    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:39:01.616578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:01.616545    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:02.403397 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:02.403357    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:04.266828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:04.266787    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:05.212566 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:05.212518    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:05.381088 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:05.381052    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:08.703560 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:08.703520    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:09.568097 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:09.568047    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:09.909568 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:09.909481    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"]
Nov 02 20:39:09.910004 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:09.909681    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" podUID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" containerName="istio-proxy" containerID="cri-o://fad2b33fbce9ece61a3c94560ca483606ff1c3c9071f84328084e64357ed72fd" gracePeriod=30
Nov 02 20:39:14.430576 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:14.430532    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:15.049060 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.049026    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"]
Nov 02 20:39:15.098752 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.098726    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39016714-nlbt9"]
Nov 02 20:39:15.170344 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.170310    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492ba5e8-9ccc-4226-805d-97a3174b27a0" path="/var/lib/kubelet/pods/492ba5e8-9ccc-4226-805d-97a3174b27a0/volumes"
Nov 02 20:39:15.194802 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.194734    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:39:15.280598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.280569    5224 generic.go:356] "Generic (PLEG): container finished" podID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" containerID="fad2b33fbce9ece61a3c94560ca483606ff1c3c9071f84328084e64357ed72fd" exitCode=0
Nov 02 20:39:15.280737 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.280613    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"
Nov 02 20:39:15.280737 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.280653    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" event={"ID":"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53","Type":"ContainerDied","Data":"fad2b33fbce9ece61a3c94560ca483606ff1c3c9071f84328084e64357ed72fd"}
Nov 02 20:39:15.280737 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.280701    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz" event={"ID":"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53","Type":"ContainerDied","Data":"f992672dba8e39371b05b32b7aa6ec2c384370ecd5552e8209ae9e3ce7a6d6f4"}
Nov 02 20:39:15.280737 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.280724    5224 scope.go:122] "RemoveContainer" containerID="fad2b33fbce9ece61a3c94560ca483606ff1c3c9071f84328084e64357ed72fd"
Nov 02 20:39:15.290292 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.290277    5224 scope.go:122] "RemoveContainer" containerID="fad2b33fbce9ece61a3c94560ca483606ff1c3c9071f84328084e64357ed72fd"
Nov 02 20:39:15.290537 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:39:15.290521    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad2b33fbce9ece61a3c94560ca483606ff1c3c9071f84328084e64357ed72fd\": container with ID starting with fad2b33fbce9ece61a3c94560ca483606ff1c3c9071f84328084e64357ed72fd not found: ID does not exist" containerID="fad2b33fbce9ece61a3c94560ca483606ff1c3c9071f84328084e64357ed72fd"
Nov 02 20:39:15.290587 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.290546    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad2b33fbce9ece61a3c94560ca483606ff1c3c9071f84328084e64357ed72fd"} err="failed to get container status \"fad2b33fbce9ece61a3c94560ca483606ff1c3c9071f84328084e64357ed72fd\": rpc error: code = NotFound desc = could not find container \"fad2b33fbce9ece61a3c94560ca483606ff1c3c9071f84328084e64357ed72fd\": container with ID starting with fad2b33fbce9ece61a3c94560ca483606ff1c3c9071f84328084e64357ed72fd not found: ID does not exist"
Nov 02 20:39:15.295911 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.295893    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-data\") pod \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") "
Nov 02 20:39:15.295976 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.295964    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-envoy\") pod \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") "
Nov 02 20:39:15.296018 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.296007    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-podinfo\") pod \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") "
Nov 02 20:39:15.296058 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.296037    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-token\" (UniqueName: \"kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-token\") pod \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") "
Nov 02 20:39:15.296111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.296082    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istiod-ca-cert\") pod \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") "
Nov 02 20:39:15.296169 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.296115    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-credential-socket\") pod \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") "
Nov 02 20:39:15.296169 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.296119    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-data" pod "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" (UID: "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:15.296169 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.296155    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-socket\") pod \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") "
Nov 02 20:39:15.296324 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.296199    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-kube-api-access-tbmgg\" (UniqueName: \"kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-kube-api-access-tbmgg\") pod \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") "
Nov 02 20:39:15.296381 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.296327    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-certs\") pod \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\" (UID: \"eecfc55c-fb22-4c8a-a63d-ecd9917c2a53\") "
Nov 02 20:39:15.296440 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.296387    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-credential-socket" pod "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" (UID: "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:15.296440 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.296400    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-socket" pod "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" (UID: "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:15.296791 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.296678    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-certs" pod "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" (UID: "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:15.296791 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.296730    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istiod-ca-cert" pod "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" (UID: "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:39:15.297012 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.296988    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:15.297102 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.297018    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:15.297102 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.297034    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:15.297102 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.297061    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:15.297102 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.297085    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:15.298318 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.298291    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-podinfo" pod "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" (UID: "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:39:15.298402 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.298372    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-envoy" pod "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" (UID: "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:15.298402 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.298382    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-token" pod "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" (UID: "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:39:15.298487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.298467    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-kube-api-access-tbmgg" pod "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" (UID: "eecfc55c-fb22-4c8a-a63d-ecd9917c2a53"). InnerVolumeSpecName "kube-api-access-tbmgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:39:15.398265 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.398237    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbmgg\" (UniqueName: \"kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-kube-api-access-tbmgg\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:15.398265 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.398265    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:15.398453 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.398275    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:15.398453 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.398283    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:15.703452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.703421    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"]
Nov 02 20:39:15.754385 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:15.754357    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kdpgz"]
Nov 02 20:39:17.175250 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:17.175211    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" path="/var/lib/kubelet/pods/eecfc55c-fb22-4c8a-a63d-ecd9917c2a53/volumes"
Nov 02 20:39:19.381573 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:19.381540    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:24.072684 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:24.072645    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:24.577534 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:24.577490    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:27.359132 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:39:27.359099    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:39:28.755051 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:28.755019    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"]
Nov 02 20:39:28.755502 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:28.755316    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9" podUID="23ca0a07-1d6a-49d1-9f58-aeef040acc4d" containerName="istio-proxy" containerID="cri-o://792962527b7abdef3dd3d60eb59561cabf0a9e19d53b1ab51b6da3456849e246" gracePeriod=30
Nov 02 20:39:28.806288 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:28.806255    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"]
Nov 02 20:39:28.806500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:28.806457    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79" podUID="e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" containerName="istio-proxy" containerID="cri-o://dea8a2cf59b74015165d659e97e14b493541b8b71260da3a11cc5a756610b5ec" gracePeriod=30
Nov 02 20:39:29.505815 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:29.505759    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:29.972082 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:29.972049    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:34.069126 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.069097    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:39:34.098223 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.098192    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:39:34.213565 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.213531    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-data\") pod \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") "
Nov 02 20:39:34.213763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.213594    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istiod-ca-cert\") pod \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") "
Nov 02 20:39:34.213763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.213656    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-certs\") pod \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") "
Nov 02 20:39:34.213763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.213708    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-socket\") pod \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") "
Nov 02 20:39:34.213763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.213747    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-token\" (UniqueName: \"kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-token\") pod \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") "
Nov 02 20:39:34.213974 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.213825    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istiod-ca-cert\") pod \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") "
Nov 02 20:39:34.213974 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.213855    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-data\") pod \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") "
Nov 02 20:39:34.213974 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.213881    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-envoy\") pod \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") "
Nov 02 20:39:34.213974 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.213922    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-socket\") pod \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") "
Nov 02 20:39:34.213974 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.213969    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-credential-socket\") pod \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") "
Nov 02 20:39:34.214170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214008    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-token\" (UniqueName: \"kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-token\") pod \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") "
Nov 02 20:39:34.214170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214038    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-credential-socket\") pod \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") "
Nov 02 20:39:34.214170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214073    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-podinfo\") pod \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") "
Nov 02 20:39:34.214170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214116    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-podinfo\") pod \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") "
Nov 02 20:39:34.214170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214166    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-certs\") pod \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") "
Nov 02 20:39:34.214443 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214210    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-kube-api-access-68pqp\" (UniqueName: \"kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-kube-api-access-68pqp\") pod \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") "
Nov 02 20:39:34.214443 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214246    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-envoy\") pod \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\" (UID: \"23ca0a07-1d6a-49d1-9f58-aeef040acc4d\") "
Nov 02 20:39:34.214443 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214294    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-kube-api-access-tq8wc\" (UniqueName: \"kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-kube-api-access-tq8wc\") pod \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\" (UID: \"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1\") "
Nov 02 20:39:34.214611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214031    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-certs" pod "23ca0a07-1d6a-49d1-9f58-aeef040acc4d" (UID: "23ca0a07-1d6a-49d1-9f58-aeef040acc4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:34.214667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214059    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-socket" pod "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" (UID: "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:34.214667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214625    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-credential-socket" pod "23ca0a07-1d6a-49d1-9f58-aeef040acc4d" (UID: "23ca0a07-1d6a-49d1-9f58-aeef040acc4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:34.214667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214100    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istiod-ca-cert" pod "23ca0a07-1d6a-49d1-9f58-aeef040acc4d" (UID: "23ca0a07-1d6a-49d1-9f58-aeef040acc4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:39:34.214667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214361    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-socket" pod "23ca0a07-1d6a-49d1-9f58-aeef040acc4d" (UID: "23ca0a07-1d6a-49d1-9f58-aeef040acc4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:34.214667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214388    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istiod-ca-cert" pod "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" (UID: "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:39:34.214667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214562    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-data" pod "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" (UID: "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:34.214667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214570    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-data" pod "23ca0a07-1d6a-49d1-9f58-aeef040acc4d" (UID: "23ca0a07-1d6a-49d1-9f58-aeef040acc4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:34.215050 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214748    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-certs" pod "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" (UID: "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:34.215050 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.214980    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-credential-socket" pod "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" (UID: "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:34.215678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.215483    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.215678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.215513    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.215678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.215529    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.215678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.215544    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.215678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.215559    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.215678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.215575    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.215678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.215591    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.215678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.215606    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.215678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.215620    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.215678 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.215634    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.217125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.217091    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-envoy" pod "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" (UID: "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:34.217218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.217161    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-token" pod "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" (UID: "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:39:34.217617 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.217587    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-kube-api-access-68pqp" pod "23ca0a07-1d6a-49d1-9f58-aeef040acc4d" (UID: "23ca0a07-1d6a-49d1-9f58-aeef040acc4d"). InnerVolumeSpecName "kube-api-access-68pqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:39:34.217684 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.217648    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-token" pod "23ca0a07-1d6a-49d1-9f58-aeef040acc4d" (UID: "23ca0a07-1d6a-49d1-9f58-aeef040acc4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:39:34.217684 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.217658    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-kube-api-access-tq8wc" pod "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" (UID: "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1"). InnerVolumeSpecName "kube-api-access-tq8wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:39:34.217818 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.217803    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-envoy" pod "23ca0a07-1d6a-49d1-9f58-aeef040acc4d" (UID: "23ca0a07-1d6a-49d1-9f58-aeef040acc4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:39:34.218332 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.218315    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-podinfo" pod "23ca0a07-1d6a-49d1-9f58-aeef040acc4d" (UID: "23ca0a07-1d6a-49d1-9f58-aeef040acc4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:39:34.218481 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.218464    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-podinfo" pod "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" (UID: "e5b61725-506b-4d0e-ab89-1cbe26f5f8c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:39:34.317302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.317214    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.317302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.317245    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.317302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.317255    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.317302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.317263    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.317302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.317272    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.317302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.317283    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68pqp\" (UniqueName: \"kubernetes.io/projected/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-kube-api-access-68pqp\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.317302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.317293    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ca0a07-1d6a-49d1-9f58-aeef040acc4d-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.317302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.317303    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq8wc\" (UniqueName: \"kubernetes.io/projected/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1-kube-api-access-tq8wc\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:39:34.366801 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.366746    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:34.395338 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.395309    5224 generic.go:356] "Generic (PLEG): container finished" podID="23ca0a07-1d6a-49d1-9f58-aeef040acc4d" containerID="792962527b7abdef3dd3d60eb59561cabf0a9e19d53b1ab51b6da3456849e246" exitCode=0
Nov 02 20:39:34.395538 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.395376    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"
Nov 02 20:39:34.395538 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.395392    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9" event={"ID":"23ca0a07-1d6a-49d1-9f58-aeef040acc4d","Type":"ContainerDied","Data":"792962527b7abdef3dd3d60eb59561cabf0a9e19d53b1ab51b6da3456849e246"}
Nov 02 20:39:34.395538 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.395454    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9" event={"ID":"23ca0a07-1d6a-49d1-9f58-aeef040acc4d","Type":"ContainerDied","Data":"102e90b314aaa500e9b55bf4486cadb042c7e66bcdc423a4ca05aafa8c7fb1a2"}
Nov 02 20:39:34.395538 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.395484    5224 scope.go:122] "RemoveContainer" containerID="792962527b7abdef3dd3d60eb59561cabf0a9e19d53b1ab51b6da3456849e246"
Nov 02 20:39:34.396903 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.396880    5224 generic.go:356] "Generic (PLEG): container finished" podID="e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" containerID="dea8a2cf59b74015165d659e97e14b493541b8b71260da3a11cc5a756610b5ec" exitCode=0
Nov 02 20:39:34.397018 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.396923    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"
Nov 02 20:39:34.397018 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.396941    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79" event={"ID":"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1","Type":"ContainerDied","Data":"dea8a2cf59b74015165d659e97e14b493541b8b71260da3a11cc5a756610b5ec"}
Nov 02 20:39:34.397018 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.396981    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79" event={"ID":"e5b61725-506b-4d0e-ab89-1cbe26f5f8c1","Type":"ContainerDied","Data":"ad094459614b77a3545e12c77f645f8cb054d0c33dcca897bcb35d1bad82d3a1"}
Nov 02 20:39:34.406665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.406633    5224 scope.go:122] "RemoveContainer" containerID="792962527b7abdef3dd3d60eb59561cabf0a9e19d53b1ab51b6da3456849e246"
Nov 02 20:39:34.406965 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:39:34.406947    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"792962527b7abdef3dd3d60eb59561cabf0a9e19d53b1ab51b6da3456849e246\": container with ID starting with 792962527b7abdef3dd3d60eb59561cabf0a9e19d53b1ab51b6da3456849e246 not found: ID does not exist" containerID="792962527b7abdef3dd3d60eb59561cabf0a9e19d53b1ab51b6da3456849e246"
Nov 02 20:39:34.407056 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.406976    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792962527b7abdef3dd3d60eb59561cabf0a9e19d53b1ab51b6da3456849e246"} err="failed to get container status \"792962527b7abdef3dd3d60eb59561cabf0a9e19d53b1ab51b6da3456849e246\": rpc error: code = NotFound desc = could not find container \"792962527b7abdef3dd3d60eb59561cabf0a9e19d53b1ab51b6da3456849e246\": container with ID starting with 792962527b7abdef3dd3d60eb59561cabf0a9e19d53b1ab51b6da3456849e246 not found: ID does not exist"
Nov 02 20:39:34.407056 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.406993    5224 scope.go:122] "RemoveContainer" containerID="dea8a2cf59b74015165d659e97e14b493541b8b71260da3a11cc5a756610b5ec"
Nov 02 20:39:34.416355 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.416334    5224 scope.go:122] "RemoveContainer" containerID="dea8a2cf59b74015165d659e97e14b493541b8b71260da3a11cc5a756610b5ec"
Nov 02 20:39:34.416656 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:39:34.416634    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea8a2cf59b74015165d659e97e14b493541b8b71260da3a11cc5a756610b5ec\": container with ID starting with dea8a2cf59b74015165d659e97e14b493541b8b71260da3a11cc5a756610b5ec not found: ID does not exist" containerID="dea8a2cf59b74015165d659e97e14b493541b8b71260da3a11cc5a756610b5ec"
Nov 02 20:39:34.416739 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.416660    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea8a2cf59b74015165d659e97e14b493541b8b71260da3a11cc5a756610b5ec"} err="failed to get container status \"dea8a2cf59b74015165d659e97e14b493541b8b71260da3a11cc5a756610b5ec\": rpc error: code = NotFound desc = could not find container \"dea8a2cf59b74015165d659e97e14b493541b8b71260da3a11cc5a756610b5ec\": container with ID starting with dea8a2cf59b74015165d659e97e14b493541b8b71260da3a11cc5a756610b5ec not found: ID does not exist"
Nov 02 20:39:34.564574 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.564539    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"]
Nov 02 20:39:34.570004 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.569946    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-tbx79"]
Nov 02 20:39:34.682639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.682599    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"]
Nov 02 20:39:34.685031 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:34.685011    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-2d7d9"]
Nov 02 20:39:35.169836 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:35.169798    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ca0a07-1d6a-49d1-9f58-aeef040acc4d" path="/var/lib/kubelet/pods/23ca0a07-1d6a-49d1-9f58-aeef040acc4d/volumes"
Nov 02 20:39:35.170649 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:35.170611    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" path="/var/lib/kubelet/pods/e5b61725-506b-4d0e-ab89-1cbe26f5f8c1/volumes"
Nov 02 20:39:39.246227 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:39.246185    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:41.006416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.006378    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"]
Nov 02 20:39:41.006917 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.006896    5224 cpu_manager.go:414] "containerMap: removing container" podUID="23ca0a07-1d6a-49d1-9f58-aeef040acc4d" containerName="istio-proxy"
Nov 02 20:39:41.006917 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.006910    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ca0a07-1d6a-49d1-9f58-aeef040acc4d" containerName="istio-proxy"
Nov 02 20:39:41.007031 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.006934    5224 cpu_manager.go:414] "containerMap: removing container" podUID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" containerName="istio-proxy"
Nov 02 20:39:41.007031 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.006940    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" containerName="istio-proxy"
Nov 02 20:39:41.007031 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.006959    5224 cpu_manager.go:414] "containerMap: removing container" podUID="e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" containerName="istio-proxy"
Nov 02 20:39:41.007031 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.006966    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" containerName="istio-proxy"
Nov 02 20:39:41.007251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.007043    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b61725-506b-4d0e-ab89-1cbe26f5f8c1" containerName="istio-proxy"
Nov 02 20:39:41.007251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.007055    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ca0a07-1d6a-49d1-9f58-aeef040acc4d" containerName="istio-proxy"
Nov 02 20:39:41.007251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.007065    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="eecfc55c-fb22-4c8a-a63d-ecd9917c2a53" containerName="istio-proxy"
Nov 02 20:39:41.011065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.011042    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.061762 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.061727    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"]
Nov 02 20:39:41.089891 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.089856    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-home\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.090054 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.089914    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-tmp-dir\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.090054 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.089981    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgf5l\" (UniqueName: \"kubernetes.io/projected/8eb533ba-79d6-45de-bb46-217781691c75-kube-api-access-cgf5l\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.090135 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.090096    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-model-cache\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.090181 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.090159    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-dshm\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.090217 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.090186    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb533ba-79d6-45de-bb46-217781691c75-tls-certs\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.110668 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.110647    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-76c687\"/\"kube-root-ca.crt\""
Nov 02 20:39:41.110668 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.110654    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-76c687\"/\"openshift-service-ca.crt\""
Nov 02 20:39:41.115690 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.115672    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-76c687\"/\"auth-test-model-76c687-kserve-self-signed-certs\""
Nov 02 20:39:41.115853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.115838    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-76c687\"/\"default-dockercfg-c8gsb\""
Nov 02 20:39:41.152156 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.152127    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:41.191872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.191840    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-home\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.192080 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.191903    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-tmp-dir\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.192080 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.191929    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgf5l\" (UniqueName: \"kubernetes.io/projected/8eb533ba-79d6-45de-bb46-217781691c75-kube-api-access-cgf5l\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.192080 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.192019    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-model-cache\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.192257 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.192085    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-dshm\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.192257 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.192118    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb533ba-79d6-45de-bb46-217781691c75-tls-certs\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.192327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.192296    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-home\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.192327 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.192315    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-tmp-dir\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.192444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.192424    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-model-cache\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.194234 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.194211    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-dshm\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.194657 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.194634    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb533ba-79d6-45de-bb46-217781691c75-tls-certs\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.250140 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.250102    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgf5l\" (UniqueName: \"kubernetes.io/projected/8eb533ba-79d6-45de-bb46-217781691c75-kube-api-access-cgf5l\") pod \"auth-test-model-76c687-kserve-55f896d9cc-2rv25\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") " pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.322491 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.322451    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:41.695057 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:39:41.695023    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8eb533ba_79d6_45de_bb46_217781691c75.slice/crio-28d6512dcca4e4873872cd15c3ff7060ea06a65409b6a7deb79fef2ad0fb6c55 WatchSource:0}: Error finding container 28d6512dcca4e4873872cd15c3ff7060ea06a65409b6a7deb79fef2ad0fb6c55: Status 404 returned error can't find the container with id 28d6512dcca4e4873872cd15c3ff7060ea06a65409b6a7deb79fef2ad0fb6c55
Nov 02 20:39:41.698810 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:41.698787    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"]
Nov 02 20:39:42.452474 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:42.452439    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25" event={"ID":"8eb533ba-79d6-45de-bb46-217781691c75","Type":"ContainerStarted","Data":"85ca7be496e64efb1bdce33ecd9769516e9922dd782b091b03ba82f0c5420366"}
Nov 02 20:39:42.452474 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:42.452485    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25" event={"ID":"8eb533ba-79d6-45de-bb46-217781691c75","Type":"ContainerStarted","Data":"28d6512dcca4e4873872cd15c3ff7060ea06a65409b6a7deb79fef2ad0fb6c55"}
Nov 02 20:39:42.570829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:42.570750    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25" podStartSLOduration=2.570735441 podStartE2EDuration="2.570735441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:39:42.570229511 +0000 UTC m=+3063.567918120" watchObservedRunningTime="2026-11-02 20:39:42.570735441 +0000 UTC m=+3063.568424047"
Nov 02 20:39:43.323515 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:43.323480    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:44.133624 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:44.133580    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:44.488753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:44.488674    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:39:49.051970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:49.051879    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:51.322795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:51.322736    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:51.335150 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:51.335127    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:51.516869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:51.516837    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:39:53.976758 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:53.976718    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:39:57.358716 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:39:57.358681    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:39:58.793608 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:39:58.793566    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:01.942504 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:01.942455    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:02.792908 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:02.792872    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:03.536893 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:03.536857    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:03.756654 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:03.756619    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:03.929933 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:03.929844    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:08.688592 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:08.688546    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:13.883604 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:13.883563    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:18.763123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:18.763078    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:19.393058 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:19.392933    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:23.561696 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:23.561657    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:40:25.486121 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:25.486079    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:27.359665 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:40:27.359627    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:40:28.659212 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:28.659157    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:33.548232 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:33.548187    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:34.531727 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:34.531690    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"]
Nov 02 20:40:34.531938 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:34.531912    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerName="istio-proxy" containerID="cri-o://06a53efd967dd0ce78402bb7877c1442d9d01b3427c4129354ca2bab1111b150" gracePeriod=30
Nov 02 20:40:34.582966 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:34.582929    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"]
Nov 02 20:40:34.583381 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:34.583139    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg" podUID="3317681e-cd5b-4c6f-a8cb-a3dec259764e" containerName="istio-proxy" containerID="cri-o://5180b194254b4ed189c14c0624397b8592d791cd963486c6b603f16cf67e8ca5" gracePeriod=30
Nov 02 20:40:38.488861 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:38.488821    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:39.797909 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.797820    5224 generic.go:356] "Generic (PLEG): container finished" podID="3317681e-cd5b-4c6f-a8cb-a3dec259764e" containerID="5180b194254b4ed189c14c0624397b8592d791cd963486c6b603f16cf67e8ca5" exitCode=0
Nov 02 20:40:39.797909 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.797889    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg" event={"ID":"3317681e-cd5b-4c6f-a8cb-a3dec259764e","Type":"ContainerDied","Data":"5180b194254b4ed189c14c0624397b8592d791cd963486c6b603f16cf67e8ca5"}
Nov 02 20:40:39.799776 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.799737    5224 generic.go:356] "Generic (PLEG): container finished" podID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerID="06a53efd967dd0ce78402bb7877c1442d9d01b3427c4129354ca2bab1111b150" exitCode=0
Nov 02 20:40:39.799981 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.799809    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" event={"ID":"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b","Type":"ContainerDied","Data":"06a53efd967dd0ce78402bb7877c1442d9d01b3427c4129354ca2bab1111b150"}
Nov 02 20:40:39.857906 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.857876    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:40:39.902992 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.902960    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:40:39.972734 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.972694    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-certs\") pod \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") "
Nov 02 20:40:39.972960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.972758    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istiod-ca-cert\") pod \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") "
Nov 02 20:40:39.972960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.972828    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-credential-socket\") pod \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") "
Nov 02 20:40:39.972960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.972871    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-socket\") pod \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") "
Nov 02 20:40:39.972960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.972896    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-credential-socket\") pod \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") "
Nov 02 20:40:39.972960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.972918    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-podinfo\") pod \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") "
Nov 02 20:40:39.972960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.972943    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-podinfo\") pod \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") "
Nov 02 20:40:39.973272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.972985    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-token\" (UniqueName: \"kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-token\") pod \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") "
Nov 02 20:40:39.973272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.973015    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-socket\") pod \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") "
Nov 02 20:40:39.973272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.973045    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-kube-api-access-sp76f\" (UniqueName: \"kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-kube-api-access-sp76f\") pod \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") "
Nov 02 20:40:39.973272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.973094    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-envoy\") pod \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") "
Nov 02 20:40:39.973272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.973153    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-envoy\") pod \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") "
Nov 02 20:40:39.973272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.973260    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-kube-api-access-km6nb\" (UniqueName: \"kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-kube-api-access-km6nb\") pod \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") "
Nov 02 20:40:39.973589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.973287    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-data\") pod \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\" (UID: \"3317681e-cd5b-4c6f-a8cb-a3dec259764e\") "
Nov 02 20:40:39.973589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.973352    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-token\" (UniqueName: \"kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-token\") pod \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") "
Nov 02 20:40:39.973589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.973385    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-certs\") pod \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") "
Nov 02 20:40:39.973589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.973370    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-certs" pod "3317681e-cd5b-4c6f-a8cb-a3dec259764e" (UID: "3317681e-cd5b-4c6f-a8cb-a3dec259764e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:39.973589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.973451    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-data\") pod \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") "
Nov 02 20:40:39.973589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.973483    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istiod-ca-cert\") pod \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\" (UID: \"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b\") "
Nov 02 20:40:39.973950 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.973864    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-socket" pod "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" (UID: "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:39.974131 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.974109    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:39.974215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.974157    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:39.974273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.974227    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istiod-ca-cert" pod "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" (UID: "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:40:39.974599 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.974570    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-credential-socket" pod "3317681e-cd5b-4c6f-a8cb-a3dec259764e" (UID: "3317681e-cd5b-4c6f-a8cb-a3dec259764e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:39.975591 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.974597    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istiod-ca-cert" pod "3317681e-cd5b-4c6f-a8cb-a3dec259764e" (UID: "3317681e-cd5b-4c6f-a8cb-a3dec259764e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:40:39.975591 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.974780    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-credential-socket" pod "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" (UID: "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:39.975591 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.974980    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-socket" pod "3317681e-cd5b-4c6f-a8cb-a3dec259764e" (UID: "3317681e-cd5b-4c6f-a8cb-a3dec259764e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:39.975591 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.975239    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-data" pod "3317681e-cd5b-4c6f-a8cb-a3dec259764e" (UID: "3317681e-cd5b-4c6f-a8cb-a3dec259764e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:39.976038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.976013    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-certs" pod "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" (UID: "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:39.977570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.977076    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-kube-api-access-sp76f" pod "3317681e-cd5b-4c6f-a8cb-a3dec259764e" (UID: "3317681e-cd5b-4c6f-a8cb-a3dec259764e"). InnerVolumeSpecName "kube-api-access-sp76f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:40:39.977570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.977305    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-data" pod "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" (UID: "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:39.977728 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.977570    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-kube-api-access-km6nb" pod "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" (UID: "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b"). InnerVolumeSpecName "kube-api-access-km6nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:40:39.978158 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.978131    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-podinfo" pod "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" (UID: "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:40:39.978964 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.978937    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-envoy" pod "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" (UID: "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:39.979946 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.979920    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-token" pod "3317681e-cd5b-4c6f-a8cb-a3dec259764e" (UID: "3317681e-cd5b-4c6f-a8cb-a3dec259764e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:40:39.980189 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.980163    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-podinfo" pod "3317681e-cd5b-4c6f-a8cb-a3dec259764e" (UID: "3317681e-cd5b-4c6f-a8cb-a3dec259764e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:40:39.982200 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.982170    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-envoy" pod "3317681e-cd5b-4c6f-a8cb-a3dec259764e" (UID: "3317681e-cd5b-4c6f-a8cb-a3dec259764e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:39.983089 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:39.983063    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-token" pod "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" (UID: "30f7c74c-68c9-4da9-b5cc-07b37f6ac76b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:40:40.075216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075123    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075151    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075162    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075172    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075181    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075197    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075207    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075215    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075223    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075232    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp76f\" (UniqueName: \"kubernetes.io/projected/3317681e-cd5b-4c6f-a8cb-a3dec259764e-kube-api-access-sp76f\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075242    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075250    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075260    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km6nb\" (UniqueName: \"kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-kube-api-access-km6nb\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075268    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3317681e-cd5b-4c6f-a8cb-a3dec259764e-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075276    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.075622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.075285    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:40.400477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.400394    5224 scope.go:122] "RemoveContainer" containerID="06a53efd967dd0ce78402bb7877c1442d9d01b3427c4129354ca2bab1111b150"
Nov 02 20:40:40.410426 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.410402    5224 scope.go:122] "RemoveContainer" containerID="9d4b83f5010bf56584d154a5beef9a4a90a361dda5a92cdd7e695cc379d00061"
Nov 02 20:40:40.419930 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.419904    5224 scope.go:122] "RemoveContainer" containerID="5180b194254b4ed189c14c0624397b8592d791cd963486c6b603f16cf67e8ca5"
Nov 02 20:40:40.429577 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.429550    5224 scope.go:122] "RemoveContainer" containerID="0741bfdc8b687d1e751b7b9c52a620d52122139299e30d1b3d7503221c5e83e3"
Nov 02 20:40:40.805508 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.805461    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg" event={"ID":"3317681e-cd5b-4c6f-a8cb-a3dec259764e","Type":"ContainerDied","Data":"08376eac095c9337fd55c77d4ae14a43d95e5ad9aaf25d7672e46afafbb8a433"}
Nov 02 20:40:40.805508 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.805494    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"
Nov 02 20:40:40.806014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.805519    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"
Nov 02 20:40:40.806014 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:40.805539    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4" event={"ID":"30f7c74c-68c9-4da9-b5cc-07b37f6ac76b","Type":"ContainerDied","Data":"abb26d0f88291e34c4b34067c412db296a9b19a7206f3c0cf65304fd33b3abf8"}
Nov 02 20:40:41.103102 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:41.103015    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"]
Nov 02 20:40:41.108546 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:41.108523    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-r45k4"]
Nov 02 20:40:41.168697 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:41.168656    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" path="/var/lib/kubelet/pods/30f7c74c-68c9-4da9-b5cc-07b37f6ac76b/volumes"
Nov 02 20:40:41.219438 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:41.219399    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"]
Nov 02 20:40:41.223140 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:41.223113    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-s24qg"]
Nov 02 20:40:41.531296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:41.531250    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:43.167479 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:43.167441    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3317681e-cd5b-4c6f-a8cb-a3dec259764e" path="/var/lib/kubelet/pods/3317681e-cd5b-4c6f-a8cb-a3dec259764e/volumes"
Nov 02 20:40:43.380641 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:43.380604    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:48.257215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:48.257177    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:53.091172 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:53.091126    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:40:54.305687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:54.305647    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:40:57.360335 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:40:57.360293    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:40:57.904544 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:40:57.904490    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:00.436181 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.436147    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"]
Nov 02 20:41:00.436712 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.436695    5224 cpu_manager.go:414] "containerMap: removing container" podUID="3317681e-cd5b-4c6f-a8cb-a3dec259764e" containerName="istio-proxy"
Nov 02 20:41:00.436752 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.436714    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3317681e-cd5b-4c6f-a8cb-a3dec259764e" containerName="istio-proxy"
Nov 02 20:41:00.436752 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.436727    5224 cpu_manager.go:414] "containerMap: removing container" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerName="istio-proxy"
Nov 02 20:41:00.436752 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.436732    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerName="istio-proxy"
Nov 02 20:41:00.436872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.436829    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3317681e-cd5b-4c6f-a8cb-a3dec259764e" containerName="istio-proxy"
Nov 02 20:41:00.436872 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.436839    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f7c74c-68c9-4da9-b5cc-07b37f6ac76b" containerName="istio-proxy"
Nov 02 20:41:00.441387 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.441366    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.493687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.493651    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"]
Nov 02 20:41:00.510733 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.510705    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnlbj\" (UniqueName: \"kubernetes.io/projected/02bb97cd-3492-4849-ae6f-3b4481e7ca42-kube-api-access-jnlbj\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.510904 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.510754    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-model-cache\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.510950 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.510927    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tmp-dir\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.510986 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.510975    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-dshm\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.511034 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.511023    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-home\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.511111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.511097    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tls-certs\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.541716 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.541684    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-09ecd0\"/\"openshift-service-ca.crt\""
Nov 02 20:41:00.541898 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.541715    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-09ecd0\"/\"kube-root-ca.crt\""
Nov 02 20:41:00.541898 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.541740    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-09ecd0\"/\"default-dockercfg-l9gt8\""
Nov 02 20:41:00.545902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.545886    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-09ecd0\"/\"auth-test-model-09ecd0-kserve-self-signed-certs\""
Nov 02 20:41:00.575927 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.575893    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:00.613897 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.613858    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-home\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.614080 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.613914    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tls-certs\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.614080 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.614016    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnlbj\" (UniqueName: \"kubernetes.io/projected/02bb97cd-3492-4849-ae6f-3b4481e7ca42-kube-api-access-jnlbj\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.614221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.614080    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-model-cache\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.614221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.614193    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tmp-dir\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.614338 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.614237    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-dshm\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.614396 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.614364    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-home\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.614485 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.614459    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-model-cache\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.614573 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.614509    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tmp-dir\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.616473 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.616440    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-dshm\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.616618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.616601    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tls-certs\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.672785 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.672678    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnlbj\" (UniqueName: \"kubernetes.io/projected/02bb97cd-3492-4849-ae6f-3b4481e7ca42-kube-api-access-jnlbj\") pod \"auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:00.755435 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:00.755398    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:01.136967 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:41:01.136938    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02bb97cd_3492_4849_ae6f_3b4481e7ca42.slice/crio-5843fe69bb7321579a111935ea81807ed72654a8abd2fdc2d5cd76d085b2be15 WatchSource:0}: Error finding container 5843fe69bb7321579a111935ea81807ed72654a8abd2fdc2d5cd76d085b2be15: Status 404 returned error can't find the container with id 5843fe69bb7321579a111935ea81807ed72654a8abd2fdc2d5cd76d085b2be15
Nov 02 20:41:01.140317 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:01.140297    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"]
Nov 02 20:41:01.932709 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:01.932666    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95" event={"ID":"02bb97cd-3492-4849-ae6f-3b4481e7ca42","Type":"ContainerStarted","Data":"259826522cad6706753e153dd43d1506021f0ce28b23a399e564ca3a31250d34"}
Nov 02 20:41:01.933143 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:01.932721    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95" event={"ID":"02bb97cd-3492-4849-ae6f-3b4481e7ca42","Type":"ContainerStarted","Data":"5843fe69bb7321579a111935ea81807ed72654a8abd2fdc2d5cd76d085b2be15"}
Nov 02 20:41:02.051753 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:02.051701    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95" podStartSLOduration=2.051689031 podStartE2EDuration="2.051689031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:41:02.05121466 +0000 UTC m=+3143.048903269" watchObservedRunningTime="2026-11-02 20:41:02.051689031 +0000 UTC m=+3143.049377639"
Nov 02 20:41:02.269702 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:02.269653    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:02.720269 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:02.720185    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:02.756113 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:02.756079    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:03.331834 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:03.331800    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:07.574354 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:07.574308    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:10.755874 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:10.755836    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:10.768448 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:10.768419    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:10.998781 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:10.998734    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:41:12.468266 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:12.468229    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:17.323281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:17.323234    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:22.200290 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:22.200200    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:24.829808 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:24.829761    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:27.360592 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:41:27.360554    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:41:27.452860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:27.452819    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:32.253438 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:32.253393    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:37.085914 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:37.085860    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:41.905025 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:41.904967    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:44.186358 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:44.186328    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:41:46.762133 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:46.762095    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:51.661337 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:51.661271    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:56.677339 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:41:56.677303    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:41:57.362488 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:41:57.362453    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:42:01.550319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:01.550244    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:02.613763 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:02.613715    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:03.414615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:03.414572    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:04.384095 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:04.384058    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:06.450538 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:06.450503    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:07.386085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:07.386038    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:09.265066 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.265029    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"]
Nov 02 20:42:09.269008 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.268984    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.321757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.321714    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"]
Nov 02 20:42:09.369661 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.369633    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6b12ae\"/\"openshift-service-ca.crt\""
Nov 02 20:42:09.369661 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.369661    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6b12ae\"/\"kube-root-ca.crt\""
Nov 02 20:42:09.373465 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.373436    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6b12ae\"/\"default-dockercfg-kxl45\""
Nov 02 20:42:09.373465 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.373459    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6b12ae\"/\"auth-test-model-6b12ae-kserve-self-signed-certs\""
Nov 02 20:42:09.392668 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.392639    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:09.392982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.392965    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-dshm\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.393038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.393017    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-tmp-dir\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.393079 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.393037    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjb7b\" (UniqueName: \"kubernetes.io/projected/34024b80-0661-4410-b654-1eaf65fe14fd-kube-api-access-qjb7b\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.393189 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.393171    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34024b80-0661-4410-b654-1eaf65fe14fd-tls-certs\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.393236 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.393226    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-home\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.393308 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.393296    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-model-cache\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.495033 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.494995    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34024b80-0661-4410-b654-1eaf65fe14fd-tls-certs\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.495294 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.495070    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-home\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.495294 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.495113    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-model-cache\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.495294 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.495205    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-dshm\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.495294 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.495265    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-tmp-dir\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.495294 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.495288    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjb7b\" (UniqueName: \"kubernetes.io/projected/34024b80-0661-4410-b654-1eaf65fe14fd-kube-api-access-qjb7b\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.495561 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.495504    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-home\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.495616 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.495570    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-model-cache\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.495671 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.495606    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-tmp-dir\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.497536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.497505    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-dshm\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.497696 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.497679    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34024b80-0661-4410-b654-1eaf65fe14fd-tls-certs\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.553295 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.553223    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjb7b\" (UniqueName: \"kubernetes.io/projected/34024b80-0661-4410-b654-1eaf65fe14fd-kube-api-access-qjb7b\") pod \"auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.581042 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.581014    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:09.948233 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:42:09.948192    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34024b80_0661_4410_b654_1eaf65fe14fd.slice/crio-8349024e625f48872cba882f74fc313cd605f5df5927b18d954d860615c62f98 WatchSource:0}: Error finding container 8349024e625f48872cba882f74fc313cd605f5df5927b18d954d860615c62f98: Status 404 returned error can't find the container with id 8349024e625f48872cba882f74fc313cd605f5df5927b18d954d860615c62f98
Nov 02 20:42:09.951988 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:09.951960    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"]
Nov 02 20:42:10.336394 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:10.336357    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs" event={"ID":"34024b80-0661-4410-b654-1eaf65fe14fd","Type":"ContainerStarted","Data":"eeb2c35cc170f840a6b2fb2ebee29f4f7886338e6d1eded08207dd5efff232f5"}
Nov 02 20:42:10.336394 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:10.336402    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs" event={"ID":"34024b80-0661-4410-b654-1eaf65fe14fd","Type":"ContainerStarted","Data":"8349024e625f48872cba882f74fc313cd605f5df5927b18d954d860615c62f98"}
Nov 02 20:42:10.453710 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:10.453649    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs" podStartSLOduration=1.453634699 podStartE2EDuration="1.453634699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:42:10.451858718 +0000 UTC m=+3211.449547327" watchObservedRunningTime="2026-11-02 20:42:10.453634699 +0000 UTC m=+3211.451323331"
Nov 02 20:42:10.581623 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:10.581591    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:11.680783 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:11.680732    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:12.925038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:12.925002    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"]
Nov 02 20:42:12.929069 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:12.929051    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:12.981338 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:12.981309    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"]
Nov 02 20:42:13.030167 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.030130    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-578636\"/\"auth-test-model-578636-kserve-self-signed-certs\""
Nov 02 20:42:13.030340 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.030195    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-578636\"/\"kube-root-ca.crt\""
Nov 02 20:42:13.030340 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.030272    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-578636\"/\"openshift-service-ca.crt\""
Nov 02 20:42:13.033444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.033426    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-578636\"/\"default-dockercfg-66vpb\""
Nov 02 20:42:13.035001 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.034969    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-home\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.035069 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.035041    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102d743c-71ea-4b15-b624-dca42b2882f9-tls-certs\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.035129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.035115    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-682rf\" (UniqueName: \"kubernetes.io/projected/102d743c-71ea-4b15-b624-dca42b2882f9-kube-api-access-682rf\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.035253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.035236    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-tmp-dir\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.035290 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.035276    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-model-cache\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.035324 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.035302    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-dshm\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.052193 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.052145    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:13.135960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.135927    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-tmp-dir\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.136136 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.135976    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-model-cache\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.136136 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.136008    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-dshm\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.136136 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.136056    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-home\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.136136 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.136132    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102d743c-71ea-4b15-b624-dca42b2882f9-tls-certs\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.136348 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.136166    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-682rf\" (UniqueName: \"kubernetes.io/projected/102d743c-71ea-4b15-b624-dca42b2882f9-kube-api-access-682rf\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.136458 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.136430    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-tmp-dir\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.136540 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.136515    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-model-cache\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.136607 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.136573    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-home\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.138428 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.138407    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-dshm\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.138728 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.138705    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102d743c-71ea-4b15-b624-dca42b2882f9-tls-certs\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.193840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.193753    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-682rf\" (UniqueName: \"kubernetes.io/projected/102d743c-71ea-4b15-b624-dca42b2882f9-kube-api-access-682rf\") pod \"auth-test-model-578636-kserve-67f69f6bbf-k8ffn\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") " pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.241320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.241279    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:13.617886 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:42:13.617842    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod102d743c_71ea_4b15_b624_dca42b2882f9.slice/crio-b0b49a5828916e6995620c39c57130f1df7a52b14585f65d8b982341726e0bef WatchSource:0}: Error finding container b0b49a5828916e6995620c39c57130f1df7a52b14585f65d8b982341726e0bef: Status 404 returned error can't find the container with id b0b49a5828916e6995620c39c57130f1df7a52b14585f65d8b982341726e0bef
Nov 02 20:42:13.621713 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:13.621687    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"]
Nov 02 20:42:14.363737 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:14.363702    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn" event={"ID":"102d743c-71ea-4b15-b624-dca42b2882f9","Type":"ContainerStarted","Data":"761291d8dcb5ae5b2e823da42247ff00613c1d7b902361492299507deffc0ff0"}
Nov 02 20:42:14.364156 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:14.363748    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn" event={"ID":"102d743c-71ea-4b15-b624-dca42b2882f9","Type":"ContainerStarted","Data":"b0b49a5828916e6995620c39c57130f1df7a52b14585f65d8b982341726e0bef"}
Nov 02 20:42:14.479688 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:14.479625    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn" podStartSLOduration=2.4796129049999998 podStartE2EDuration="2.479612905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:42:14.478425304 +0000 UTC m=+3215.476113912" watchObservedRunningTime="2026-11-02 20:42:14.479612905 +0000 UTC m=+3215.477301550"
Nov 02 20:42:15.242111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:15.242078    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:16.490325 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:16.490289    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:17.811026 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:17.810992    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:18.410946 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:18.410911    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:19.581544 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:19.581513    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:19.594266 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:19.594238    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:20.410663 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:20.410634    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:42:21.478970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:21.478934    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:23.241990 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:23.241957    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:23.257302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:23.257274    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:23.426856 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:23.426826    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:42:24.929380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:24.929346    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:42:26.368351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:26.368203    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:27.358385 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:42:27.358349    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:42:31.277204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:31.277165    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:36.156306 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:36.156250    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:41.039181 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:41.039125    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:45.898840 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:45.898789    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:50.681284 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:50.681247    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:51.485049 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:51.485020    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"]
Nov 02 20:42:51.485285 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:51.485262    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" containerName="istio-proxy" containerID="cri-o://0dda4596c0cc14bb5766d11d6a3cafae5749a02898b44f30e558feeda315c0a9" gracePeriod=30
Nov 02 20:42:53.028937 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:53.028898    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:53.494580 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:53.494493    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:55.548017 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:55.547972    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:56.618608 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.618576    5224 generic.go:356] "Generic (PLEG): container finished" podID="11fd9fb0-acae-4989-be05-0c80707435ed" containerID="0dda4596c0cc14bb5766d11d6a3cafae5749a02898b44f30e558feeda315c0a9" exitCode=0
Nov 02 20:42:56.619058 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.618648    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" event={"ID":"11fd9fb0-acae-4989-be05-0c80707435ed","Type":"ContainerDied","Data":"0dda4596c0cc14bb5766d11d6a3cafae5749a02898b44f30e558feeda315c0a9"}
Nov 02 20:42:56.619058 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.618728    5224 scope.go:122] "RemoveContainer" containerID="5076ceea02f90b6461c786511cc1fe4b576f9ebc42e919ac8d2d81aa3a16ab22"
Nov 02 20:42:56.748800 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.748717    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:42:56.829107 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.829074    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11fd9fb0-acae-4989-be05-0c80707435ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fd9fb0-acae-4989-be05-0c80707435ed-istio-podinfo\") pod \"11fd9fb0-acae-4989-be05-0c80707435ed\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") "
Nov 02 20:42:56.829107 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.829111    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-kube-api-access-sq4bw\" (UniqueName: \"kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-kube-api-access-sq4bw\") pod \"11fd9fb0-acae-4989-be05-0c80707435ed\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") "
Nov 02 20:42:56.829349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.829248    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-credential-socket\") pod \"11fd9fb0-acae-4989-be05-0c80707435ed\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") "
Nov 02 20:42:56.829415 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.829377    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-certs\") pod \"11fd9fb0-acae-4989-be05-0c80707435ed\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") "
Nov 02 20:42:56.829479 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.829458    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-istio-token\" (UniqueName: \"kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-istio-token\") pod \"11fd9fb0-acae-4989-be05-0c80707435ed\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") "
Nov 02 20:42:56.829546 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.829508    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-data\") pod \"11fd9fb0-acae-4989-be05-0c80707435ed\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") "
Nov 02 20:42:56.829605 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.829557    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11fd9fb0-acae-4989-be05-0c80707435ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fd9fb0-acae-4989-be05-0c80707435ed-istiod-ca-cert\") pod \"11fd9fb0-acae-4989-be05-0c80707435ed\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") "
Nov 02 20:42:56.829665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.829616    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-envoy\") pod \"11fd9fb0-acae-4989-be05-0c80707435ed\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") "
Nov 02 20:42:56.829665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.829642    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-socket\") pod \"11fd9fb0-acae-4989-be05-0c80707435ed\" (UID: \"11fd9fb0-acae-4989-be05-0c80707435ed\") "
Nov 02 20:42:56.830011 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.829589    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-credential-socket" pod "11fd9fb0-acae-4989-be05-0c80707435ed" (UID: "11fd9fb0-acae-4989-be05-0c80707435ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:56.830011 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.829678    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-certs" pod "11fd9fb0-acae-4989-be05-0c80707435ed" (UID: "11fd9fb0-acae-4989-be05-0c80707435ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:56.830011 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.829824    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-data" pod "11fd9fb0-acae-4989-be05-0c80707435ed" (UID: "11fd9fb0-acae-4989-be05-0c80707435ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:56.830289 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.830228    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:56.830289 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.830238    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-socket" pod "11fd9fb0-acae-4989-be05-0c80707435ed" (UID: "11fd9fb0-acae-4989-be05-0c80707435ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:56.830289 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.830250    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:56.830289 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.830268    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:56.830510 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.830362    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11fd9fb0-acae-4989-be05-0c80707435ed-istiod-ca-cert" pod "11fd9fb0-acae-4989-be05-0c80707435ed" (UID: "11fd9fb0-acae-4989-be05-0c80707435ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:42:56.831612 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.831583    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-kube-api-access-sq4bw" pod "11fd9fb0-acae-4989-be05-0c80707435ed" (UID: "11fd9fb0-acae-4989-be05-0c80707435ed"). InnerVolumeSpecName "kube-api-access-sq4bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:42:56.831887 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.831863    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11fd9fb0-acae-4989-be05-0c80707435ed-istio-podinfo" pod "11fd9fb0-acae-4989-be05-0c80707435ed" (UID: "11fd9fb0-acae-4989-be05-0c80707435ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:42:56.832220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.832201    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-envoy" pod "11fd9fb0-acae-4989-be05-0c80707435ed" (UID: "11fd9fb0-acae-4989-be05-0c80707435ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:56.832437 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.832421    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-istio-token" pod "11fd9fb0-acae-4989-be05-0c80707435ed" (UID: "11fd9fb0-acae-4989-be05-0c80707435ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:42:56.931125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.931096    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fd9fb0-acae-4989-be05-0c80707435ed-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:56.931125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.931120    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:56.931125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.931130    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fd9fb0-acae-4989-be05-0c80707435ed-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:56.931362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.931140    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fd9fb0-acae-4989-be05-0c80707435ed-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:56.931362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.931158    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq4bw\" (UniqueName: \"kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-kube-api-access-sq4bw\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:56.931362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:56.931168    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fd9fb0-acae-4989-be05-0c80707435ed-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:57.358860 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:42:57.358826    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:42:57.625863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:57.625817    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" event={"ID":"11fd9fb0-acae-4989-be05-0c80707435ed","Type":"ContainerDied","Data":"a951dc096ef208fd164423d775429dd0337f3685f73f37b3e069531b4ee17954"}
Nov 02 20:42:57.626330 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:57.625884    5224 scope.go:122] "RemoveContainer" containerID="0dda4596c0cc14bb5766d11d6a3cafae5749a02898b44f30e558feeda315c0a9"
Nov 02 20:42:57.626330 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:57.625832    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"
Nov 02 20:42:57.698150 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:57.698121    5224 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:42:57.698319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:57.698171    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:42:57.794283 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:57.794251    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"]
Nov 02 20:42:57.806071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:57.806043    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-wjxk7"]
Nov 02 20:42:58.507855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:58.507820    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:42:59.168311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:42:59.168281    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" path="/var/lib/kubelet/pods/11fd9fb0-acae-4989-be05-0c80707435ed/volumes"
Nov 02 20:43:00.470211 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:00.470159    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:02.950082 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:02.950044    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:03.731240 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:03.731201    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:05.343960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:05.343915    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:10.166893 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:10.166855    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:15.146176 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:15.146088    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:17.818985 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:17.818949    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:19.046176 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:19.046135    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:19.262675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:19.262638    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:20.302137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:20.302101    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:23.260925 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:23.260894    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:25.374640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:25.374601    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:27.358435 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:43:27.358401    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:43:30.249843 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:30.249805    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:35.176484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:35.176446    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:40.049279 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:40.049227    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:44.299490 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:44.299449    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:46.088953 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:46.088917    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:46.266826 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:46.266749    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:47.714684 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:47.714647    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:48.310085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:48.310048    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:48.895372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:48.895327    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:49.455194 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:49.455150    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:49.837900 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:49.837859    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:51.426050 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:51.426013    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:51.733117 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:51.733031    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:51.911406 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:51.911369    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:52.099632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:52.099594    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:56.449505 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:43:56.449468    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:43:57.358658 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:43:57.358619    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:44:00.079995 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:00.079954    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:01.358490 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:01.358448    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:03.272619 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:03.272583    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:04.280988 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:04.280937    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:06.164354 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:06.164322    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:06.695234 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:06.695201    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:44:07.248273 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:07.248240    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:08.308291 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:08.308256    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:09.991030 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:09.990998    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"]
Nov 02 20:44:09.991547 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:09.991227    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq" podUID="389ac8d7-6a0d-4007-a07d-d0851b28ab5e" containerName="istio-proxy" containerID="cri-o://bdbb882e99d107f61054f4cfa7e9aa42c5c69da135a49fedf3372f30f027e48b" gracePeriod=30
Nov 02 20:44:11.052541 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:11.052508    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:15.250000 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.249973    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:44:15.422059 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422025    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-credential-socket\") pod \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") "
Nov 02 20:44:15.422252 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422154    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-data\") pod \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") "
Nov 02 20:44:15.422252 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422189    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-token\" (UniqueName: \"kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-token\") pod \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") "
Nov 02 20:44:15.422383 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422261    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istiod-ca-cert\") pod \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") "
Nov 02 20:44:15.422383 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422318    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-podinfo\") pod \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") "
Nov 02 20:44:15.422383 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422344    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-credential-socket" pod "389ac8d7-6a0d-4007-a07d-d0851b28ab5e" (UID: "389ac8d7-6a0d-4007-a07d-d0851b28ab5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:44:15.422383 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422356    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-envoy\") pod \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") "
Nov 02 20:44:15.422610 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422420    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-kube-api-access-wmn6p\" (UniqueName: \"kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-kube-api-access-wmn6p\") pod \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") "
Nov 02 20:44:15.422610 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422430    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-data" pod "389ac8d7-6a0d-4007-a07d-d0851b28ab5e" (UID: "389ac8d7-6a0d-4007-a07d-d0851b28ab5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:44:15.422610 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422455    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-socket\") pod \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") "
Nov 02 20:44:15.422610 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422484    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-certs\") pod \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\" (UID: \"389ac8d7-6a0d-4007-a07d-d0851b28ab5e\") "
Nov 02 20:44:15.422992 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422900    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-certs" pod "389ac8d7-6a0d-4007-a07d-d0851b28ab5e" (UID: "389ac8d7-6a0d-4007-a07d-d0851b28ab5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:44:15.422992 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422961    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-socket" pod "389ac8d7-6a0d-4007-a07d-d0851b28ab5e" (UID: "389ac8d7-6a0d-4007-a07d-d0851b28ab5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:44:15.422992 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422975    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:15.423259 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.422998    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:15.423636 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.423600    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istiod-ca-cert" pod "389ac8d7-6a0d-4007-a07d-d0851b28ab5e" (UID: "389ac8d7-6a0d-4007-a07d-d0851b28ab5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:44:15.424534 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.424503    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-podinfo" pod "389ac8d7-6a0d-4007-a07d-d0851b28ab5e" (UID: "389ac8d7-6a0d-4007-a07d-d0851b28ab5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:44:15.424687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.424659    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-kube-api-access-wmn6p" pod "389ac8d7-6a0d-4007-a07d-d0851b28ab5e" (UID: "389ac8d7-6a0d-4007-a07d-d0851b28ab5e"). InnerVolumeSpecName "kube-api-access-wmn6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:44:15.424796 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.424724    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-token" pod "389ac8d7-6a0d-4007-a07d-d0851b28ab5e" (UID: "389ac8d7-6a0d-4007-a07d-d0851b28ab5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:44:15.424908 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.424885    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-envoy" pod "389ac8d7-6a0d-4007-a07d-d0851b28ab5e" (UID: "389ac8d7-6a0d-4007-a07d-d0851b28ab5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:44:15.524134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.524104    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:15.524134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.524132    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:15.524134 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.524141    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:15.524380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.524150    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmn6p\" (UniqueName: \"kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-kube-api-access-wmn6p\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:15.524380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.524161    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:15.524380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.524170    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:15.524380 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.524178    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389ac8d7-6a0d-4007-a07d-d0851b28ab5e-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:15.947923 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:15.947890    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:16.076994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:16.076962    5224 generic.go:356] "Generic (PLEG): container finished" podID="389ac8d7-6a0d-4007-a07d-d0851b28ab5e" containerID="bdbb882e99d107f61054f4cfa7e9aa42c5c69da135a49fedf3372f30f027e48b" exitCode=0
Nov 02 20:44:16.077191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:16.077009    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"
Nov 02 20:44:16.077191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:16.077047    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq" event={"ID":"389ac8d7-6a0d-4007-a07d-d0851b28ab5e","Type":"ContainerDied","Data":"bdbb882e99d107f61054f4cfa7e9aa42c5c69da135a49fedf3372f30f027e48b"}
Nov 02 20:44:16.077191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:16.077112    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq" event={"ID":"389ac8d7-6a0d-4007-a07d-d0851b28ab5e","Type":"ContainerDied","Data":"46d10816198212ccd89fa446a26aa56fc9533f3e1282755a7e2fb755054df305"}
Nov 02 20:44:16.077191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:16.077142    5224 scope.go:122] "RemoveContainer" containerID="bdbb882e99d107f61054f4cfa7e9aa42c5c69da135a49fedf3372f30f027e48b"
Nov 02 20:44:16.088321 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:16.088297    5224 scope.go:122] "RemoveContainer" containerID="bdbb882e99d107f61054f4cfa7e9aa42c5c69da135a49fedf3372f30f027e48b"
Nov 02 20:44:16.088656 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:44:16.088634    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdbb882e99d107f61054f4cfa7e9aa42c5c69da135a49fedf3372f30f027e48b\": container with ID starting with bdbb882e99d107f61054f4cfa7e9aa42c5c69da135a49fedf3372f30f027e48b not found: ID does not exist" containerID="bdbb882e99d107f61054f4cfa7e9aa42c5c69da135a49fedf3372f30f027e48b"
Nov 02 20:44:16.088734 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:16.088667    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdbb882e99d107f61054f4cfa7e9aa42c5c69da135a49fedf3372f30f027e48b"} err="failed to get container status \"bdbb882e99d107f61054f4cfa7e9aa42c5c69da135a49fedf3372f30f027e48b\": rpc error: code = NotFound desc = could not find container \"bdbb882e99d107f61054f4cfa7e9aa42c5c69da135a49fedf3372f30f027e48b\": container with ID starting with bdbb882e99d107f61054f4cfa7e9aa42c5c69da135a49fedf3372f30f027e48b not found: ID does not exist"
Nov 02 20:44:16.246536 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:16.246452    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"]
Nov 02 20:44:16.272808 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:16.272759    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-xrpkq"]
Nov 02 20:44:17.167978 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:17.167945    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389ac8d7-6a0d-4007-a07d-d0851b28ab5e" path="/var/lib/kubelet/pods/389ac8d7-6a0d-4007-a07d-d0851b28ab5e/volumes"
Nov 02 20:44:17.560127 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:17.560087    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:20.846904 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:20.846816    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:25.825368 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:25.825326    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:27.358600 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:44:27.358568    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:44:30.687368 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:30.687329    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:35.576322 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:35.576279    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:40.462243 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:40.462199    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:44.538584 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:44.538542    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:44.836841 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:44.836725    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:45.007064 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:45.007025    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:45.497628 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:45.497585    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:50.006454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:50.006412    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:50.357621 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:50.357585    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:55.240783 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:44:55.240723    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:44:57.358157 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:44:57.358118    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:45:00.054125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.054089    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:00.190056 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.190020    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"]
Nov 02 20:45:00.190525 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.190512    5224 cpu_manager.go:414] "containerMap: removing container" podUID="389ac8d7-6a0d-4007-a07d-d0851b28ab5e" containerName="istio-proxy"
Nov 02 20:45:00.190571 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.190527    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389ac8d7-6a0d-4007-a07d-d0851b28ab5e" containerName="istio-proxy"
Nov 02 20:45:00.190571 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.190548    5224 cpu_manager.go:414] "containerMap: removing container" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" containerName="istio-proxy"
Nov 02 20:45:00.190571 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.190554    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" containerName="istio-proxy"
Nov 02 20:45:00.190671 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.190574    5224 cpu_manager.go:414] "containerMap: removing container" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" containerName="istio-proxy"
Nov 02 20:45:00.190671 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.190581    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" containerName="istio-proxy"
Nov 02 20:45:00.190734 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.190674    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="389ac8d7-6a0d-4007-a07d-d0851b28ab5e" containerName="istio-proxy"
Nov 02 20:45:00.190734 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.190685    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" containerName="istio-proxy"
Nov 02 20:45:00.190734 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.190692    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="11fd9fb0-acae-4989-be05-0c80707435ed" containerName="istio-proxy"
Nov 02 20:45:00.195406 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.195379    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"]
Nov 02 20:45:00.195573 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.195513    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"
Nov 02 20:45:00.199392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.199373    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"]
Nov 02 20:45:00.199516 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.199492    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"
Nov 02 20:45:00.202973 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.202952    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"
Nov 02 20:45:00.211388 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.211368    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxs8h\" (UniqueName: \"kubernetes.io/projected/31f0697b-2658-41aa-a282-31f018b79e86-kube-api-access-mxs8h\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5\" (UID: \"31f0697b-2658-41aa-a282-31f018b79e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"
Nov 02 20:45:00.211512 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.211492    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f0697b-2658-41aa-a282-31f018b79e86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5\" (UID: \"31f0697b-2658-41aa-a282-31f018b79e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"
Nov 02 20:45:00.241282 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.241261    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"]
Nov 02 20:45:00.247117 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.247101    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"
Nov 02 20:45:00.310983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.310907    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"]
Nov 02 20:45:00.312391 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.312360    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"]
Nov 02 20:45:00.313846 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.313819    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxs8h\" (UniqueName: \"kubernetes.io/projected/31f0697b-2658-41aa-a282-31f018b79e86-kube-api-access-mxs8h\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5\" (UID: \"31f0697b-2658-41aa-a282-31f018b79e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"
Nov 02 20:45:00.313982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.313896    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2882985f-334d-4bf0-a48c-1fd5af4e70e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj\" (UID: \"2882985f-334d-4bf0-a48c-1fd5af4e70e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"
Nov 02 20:45:00.313982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.313941    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfjq4\" (UniqueName: \"kubernetes.io/projected/8dca96fd-6559-47bf-bc64-f40571e6862c-kube-api-access-pfjq4\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x\" (UID: \"8dca96fd-6559-47bf-bc64-f40571e6862c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"
Nov 02 20:45:00.314106 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.314006    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f0697b-2658-41aa-a282-31f018b79e86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5\" (UID: \"31f0697b-2658-41aa-a282-31f018b79e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"
Nov 02 20:45:00.314106 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.314077    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qggxp\" (UniqueName: \"kubernetes.io/projected/2882985f-334d-4bf0-a48c-1fd5af4e70e2-kube-api-access-qggxp\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj\" (UID: \"2882985f-334d-4bf0-a48c-1fd5af4e70e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"
Nov 02 20:45:00.314218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.314147    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dca96fd-6559-47bf-bc64-f40571e6862c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x\" (UID: \"8dca96fd-6559-47bf-bc64-f40571e6862c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"
Nov 02 20:45:00.314218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.314196    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnbwc\" (UniqueName: \"kubernetes.io/projected/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-kube-api-access-cnbwc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt\" (UID: \"9bfed799-f1b8-4f33-97fb-a7d7d19313a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"
Nov 02 20:45:00.314335 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.314227    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"]
Nov 02 20:45:00.314335 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.314254    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt\" (UID: \"9bfed799-f1b8-4f33-97fb-a7d7d19313a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"
Nov 02 20:45:00.315254 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.315233    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f0697b-2658-41aa-a282-31f018b79e86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5\" (UID: \"31f0697b-2658-41aa-a282-31f018b79e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"
Nov 02 20:45:00.316877 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.316853    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"]
Nov 02 20:45:00.356478 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.356452    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 20:45:00.372012 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.371984    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxs8h\" (UniqueName: \"kubernetes.io/projected/31f0697b-2658-41aa-a282-31f018b79e86-kube-api-access-mxs8h\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5\" (UID: \"31f0697b-2658-41aa-a282-31f018b79e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"
Nov 02 20:45:00.415056 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.415024    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt\" (UID: \"9bfed799-f1b8-4f33-97fb-a7d7d19313a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"
Nov 02 20:45:00.415195 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.415098    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2882985f-334d-4bf0-a48c-1fd5af4e70e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj\" (UID: \"2882985f-334d-4bf0-a48c-1fd5af4e70e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"
Nov 02 20:45:00.415195 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.415134    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfjq4\" (UniqueName: \"kubernetes.io/projected/8dca96fd-6559-47bf-bc64-f40571e6862c-kube-api-access-pfjq4\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x\" (UID: \"8dca96fd-6559-47bf-bc64-f40571e6862c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"
Nov 02 20:45:00.415310 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.415250    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qggxp\" (UniqueName: \"kubernetes.io/projected/2882985f-334d-4bf0-a48c-1fd5af4e70e2-kube-api-access-qggxp\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj\" (UID: \"2882985f-334d-4bf0-a48c-1fd5af4e70e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"
Nov 02 20:45:00.415365 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.415323    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dca96fd-6559-47bf-bc64-f40571e6862c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x\" (UID: \"8dca96fd-6559-47bf-bc64-f40571e6862c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"
Nov 02 20:45:00.415426 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.415372    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnbwc\" (UniqueName: \"kubernetes.io/projected/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-kube-api-access-cnbwc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt\" (UID: \"9bfed799-f1b8-4f33-97fb-a7d7d19313a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"
Nov 02 20:45:00.415959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.415852    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt\" (UID: \"9bfed799-f1b8-4f33-97fb-a7d7d19313a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"
Nov 02 20:45:00.415959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.415884    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2882985f-334d-4bf0-a48c-1fd5af4e70e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj\" (UID: \"2882985f-334d-4bf0-a48c-1fd5af4e70e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"
Nov 02 20:45:00.416125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.416098    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dca96fd-6559-47bf-bc64-f40571e6862c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x\" (UID: \"8dca96fd-6559-47bf-bc64-f40571e6862c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"
Nov 02 20:45:00.473171 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.473141    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnbwc\" (UniqueName: \"kubernetes.io/projected/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-kube-api-access-cnbwc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt\" (UID: \"9bfed799-f1b8-4f33-97fb-a7d7d19313a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"
Nov 02 20:45:00.473171 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.473167    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfjq4\" (UniqueName: \"kubernetes.io/projected/8dca96fd-6559-47bf-bc64-f40571e6862c-kube-api-access-pfjq4\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x\" (UID: \"8dca96fd-6559-47bf-bc64-f40571e6862c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"
Nov 02 20:45:00.473336 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.473239    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qggxp\" (UniqueName: \"kubernetes.io/projected/2882985f-334d-4bf0-a48c-1fd5af4e70e2-kube-api-access-qggxp\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj\" (UID: \"2882985f-334d-4bf0-a48c-1fd5af4e70e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"
Nov 02 20:45:00.510811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.510789    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"
Nov 02 20:45:00.517725 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.517700    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"
Nov 02 20:45:00.523711 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.523690    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"
Nov 02 20:45:00.560101 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.560070    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"
Nov 02 20:45:00.891327 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:45:00.891296    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f0697b_2658_41aa_a282_31f018b79e86.slice/crio-0c93a7036abbdc020635a7e820794c631da126751cecc6d4b941cbcc644303d0 WatchSource:0}: Error finding container 0c93a7036abbdc020635a7e820794c631da126751cecc6d4b941cbcc644303d0: Status 404 returned error can't find the container with id 0c93a7036abbdc020635a7e820794c631da126751cecc6d4b941cbcc644303d0
Nov 02 20:45:00.894713 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.894691    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"]
Nov 02 20:45:00.938705 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:45:00.938679    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2882985f_334d_4bf0_a48c_1fd5af4e70e2.slice/crio-af4965827e97630cba61d77d6cfba4a1f74f232245bf28d344d7bc315c777e63 WatchSource:0}: Error finding container af4965827e97630cba61d77d6cfba4a1f74f232245bf28d344d7bc315c777e63: Status 404 returned error can't find the container with id af4965827e97630cba61d77d6cfba4a1f74f232245bf28d344d7bc315c777e63
Nov 02 20:45:00.941198 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:45:00.941174    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bfed799_f1b8_4f33_97fb_a7d7d19313a2.slice/crio-8c73abcdac9e628559e3f13c4421ffcc51406a600c1f49dbb39515a7050c1ccc WatchSource:0}: Error finding container 8c73abcdac9e628559e3f13c4421ffcc51406a600c1f49dbb39515a7050c1ccc: Status 404 returned error can't find the container with id 8c73abcdac9e628559e3f13c4421ffcc51406a600c1f49dbb39515a7050c1ccc
Nov 02 20:45:00.942904 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.942885    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"]
Nov 02 20:45:00.945201 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.945161    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"]
Nov 02 20:45:00.986681 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:45:00.986658    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dca96fd_6559_47bf_bc64_f40571e6862c.slice/crio-7d797b08949a08d15bc57cfcaa24809f26f73230a914c0e9ccd83deb3f7414d0 WatchSource:0}: Error finding container 7d797b08949a08d15bc57cfcaa24809f26f73230a914c0e9ccd83deb3f7414d0: Status 404 returned error can't find the container with id 7d797b08949a08d15bc57cfcaa24809f26f73230a914c0e9ccd83deb3f7414d0
Nov 02 20:45:00.990787 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:00.990757    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"]
Nov 02 20:45:01.287362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:01.287327    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:45:01.351605 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:01.351561    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5" event={"ID":"31f0697b-2658-41aa-a282-31f018b79e86","Type":"ContainerStarted","Data":"d73e7e7a707c67d2a728ed8b3b126872a736435168aa8ef34a0b64c00d323a78"}
Nov 02 20:45:01.351812 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:01.351627    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5" event={"ID":"31f0697b-2658-41aa-a282-31f018b79e86","Type":"ContainerStarted","Data":"0c93a7036abbdc020635a7e820794c631da126751cecc6d4b941cbcc644303d0"}
Nov 02 20:45:01.353290 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:01.353266    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x" event={"ID":"8dca96fd-6559-47bf-bc64-f40571e6862c","Type":"ContainerStarted","Data":"7d797b08949a08d15bc57cfcaa24809f26f73230a914c0e9ccd83deb3f7414d0"}
Nov 02 20:45:01.354628 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:01.354605    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt" event={"ID":"9bfed799-f1b8-4f33-97fb-a7d7d19313a2","Type":"ContainerStarted","Data":"8c73abcdac9e628559e3f13c4421ffcc51406a600c1f49dbb39515a7050c1ccc"}
Nov 02 20:45:01.356018 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:01.355997    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj" event={"ID":"2882985f-334d-4bf0-a48c-1fd5af4e70e2","Type":"ContainerStarted","Data":"af4965827e97630cba61d77d6cfba4a1f74f232245bf28d344d7bc315c777e63"}
Nov 02 20:45:01.470247 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:01.470155    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5" podStartSLOduration=1.470140783 podStartE2EDuration="1.470140783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:01.467723028 +0000 UTC m=+3382.465411649" watchObservedRunningTime="2026-11-02 20:45:01.470140783 +0000 UTC m=+3382.467829391"
Nov 02 20:45:02.362988 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:02.362908    5224 generic.go:356] "Generic (PLEG): container finished" podID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2" containerID="6459bcabd901c08a7f7a7f7571b4a3d543c456f11c56e39f51e7224b2b3d3a23" exitCode=60
Nov 02 20:45:02.362988 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:02.362979    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt" event={"ID":"9bfed799-f1b8-4f33-97fb-a7d7d19313a2","Type":"ContainerDied","Data":"6459bcabd901c08a7f7a7f7571b4a3d543c456f11c56e39f51e7224b2b3d3a23"}
Nov 02 20:45:02.363452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:02.363216    5224 scope.go:122] "RemoveContainer" containerID="6459bcabd901c08a7f7a7f7571b4a3d543c456f11c56e39f51e7224b2b3d3a23"
Nov 02 20:45:02.364665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:02.364638    5224 generic.go:356] "Generic (PLEG): container finished" podID="2882985f-334d-4bf0-a48c-1fd5af4e70e2" containerID="f1e446a6895f12bf05855fb9acc92c808cf7ebb53ab3875212507aaec80987bf" exitCode=60
Nov 02 20:45:02.364826 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:02.364712    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj" event={"ID":"2882985f-334d-4bf0-a48c-1fd5af4e70e2","Type":"ContainerDied","Data":"f1e446a6895f12bf05855fb9acc92c808cf7ebb53ab3875212507aaec80987bf"}
Nov 02 20:45:02.364963 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:02.364944    5224 scope.go:122] "RemoveContainer" containerID="f1e446a6895f12bf05855fb9acc92c808cf7ebb53ab3875212507aaec80987bf"
Nov 02 20:45:02.366479 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:02.366413    5224 generic.go:356] "Generic (PLEG): container finished" podID="31f0697b-2658-41aa-a282-31f018b79e86" containerID="d73e7e7a707c67d2a728ed8b3b126872a736435168aa8ef34a0b64c00d323a78" exitCode=60
Nov 02 20:45:02.366539 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:02.366491    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5" event={"ID":"31f0697b-2658-41aa-a282-31f018b79e86","Type":"ContainerDied","Data":"d73e7e7a707c67d2a728ed8b3b126872a736435168aa8ef34a0b64c00d323a78"}
Nov 02 20:45:02.366704 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:02.366679    5224 scope.go:122] "RemoveContainer" containerID="d73e7e7a707c67d2a728ed8b3b126872a736435168aa8ef34a0b64c00d323a78"
Nov 02 20:45:02.368160 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:02.368143    5224 generic.go:356] "Generic (PLEG): container finished" podID="8dca96fd-6559-47bf-bc64-f40571e6862c" containerID="ad36a9eef5028113bedd79c456583fdc8ca01ff7a72937124c7297825d7179d5" exitCode=60
Nov 02 20:45:02.368218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:02.368202    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x" event={"ID":"8dca96fd-6559-47bf-bc64-f40571e6862c","Type":"ContainerDied","Data":"ad36a9eef5028113bedd79c456583fdc8ca01ff7a72937124c7297825d7179d5"}
Nov 02 20:45:02.368375 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:02.368361    5224 scope.go:122] "RemoveContainer" containerID="ad36a9eef5028113bedd79c456583fdc8ca01ff7a72937124c7297825d7179d5"
Nov 02 20:45:03.375733 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.375698    5224 generic.go:356] "Generic (PLEG): container finished" podID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2" containerID="5735b42745c43fd2ae11befdb54755bc5e16871876de14ed0034f217d7179bc6" exitCode=60
Nov 02 20:45:03.376168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.375785    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt" event={"ID":"9bfed799-f1b8-4f33-97fb-a7d7d19313a2","Type":"ContainerDied","Data":"5735b42745c43fd2ae11befdb54755bc5e16871876de14ed0034f217d7179bc6"}
Nov 02 20:45:03.376168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.375855    5224 scope.go:122] "RemoveContainer" containerID="6459bcabd901c08a7f7a7f7571b4a3d543c456f11c56e39f51e7224b2b3d3a23"
Nov 02 20:45:03.376168 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.376143    5224 scope.go:122] "RemoveContainer" containerID="5735b42745c43fd2ae11befdb54755bc5e16871876de14ed0034f217d7179bc6"
Nov 02 20:45:03.376428 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:03.376410    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt_odh-ai-gateway-infra(9bfed799-f1b8-4f33-97fb-a7d7d19313a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt" podUID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2"
Nov 02 20:45:03.378450 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.378421    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj" event={"ID":"2882985f-334d-4bf0-a48c-1fd5af4e70e2","Type":"ContainerStarted","Data":"3ae18dac3e41ccb519ed1b8d978a0014297e4f39c14c598b299ffc09fb0aec0b"}
Nov 02 20:45:03.380510 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.380487    5224 generic.go:356] "Generic (PLEG): container finished" podID="31f0697b-2658-41aa-a282-31f018b79e86" containerID="a9939281b062a051ae1611b46ed51ca6c9b1bc2552763ecee7b50b8ddc780509" exitCode=60
Nov 02 20:45:03.380655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.380578    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5" event={"ID":"31f0697b-2658-41aa-a282-31f018b79e86","Type":"ContainerDied","Data":"a9939281b062a051ae1611b46ed51ca6c9b1bc2552763ecee7b50b8ddc780509"}
Nov 02 20:45:03.380948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.380927    5224 scope.go:122] "RemoveContainer" containerID="a9939281b062a051ae1611b46ed51ca6c9b1bc2552763ecee7b50b8ddc780509"
Nov 02 20:45:03.381247 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:03.381222    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5_odh-ai-gateway-infra(31f0697b-2658-41aa-a282-31f018b79e86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5" podUID="31f0697b-2658-41aa-a282-31f018b79e86"
Nov 02 20:45:03.382817 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.382755    5224 generic.go:356] "Generic (PLEG): container finished" podID="8dca96fd-6559-47bf-bc64-f40571e6862c" containerID="d24b7ee1feedbfbc253253ec73f03a791cf75bb84177ed6dc0048d5fab389307" exitCode=60
Nov 02 20:45:03.382817 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.382806    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x" event={"ID":"8dca96fd-6559-47bf-bc64-f40571e6862c","Type":"ContainerDied","Data":"d24b7ee1feedbfbc253253ec73f03a791cf75bb84177ed6dc0048d5fab389307"}
Nov 02 20:45:03.383093 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.383077    5224 scope.go:122] "RemoveContainer" containerID="d24b7ee1feedbfbc253253ec73f03a791cf75bb84177ed6dc0048d5fab389307"
Nov 02 20:45:03.383360 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:03.383339    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x_odh-ai-gateway-infra(8dca96fd-6559-47bf-bc64-f40571e6862c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x" podUID="8dca96fd-6559-47bf-bc64-f40571e6862c"
Nov 02 20:45:03.519413 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.519386    5224 scope.go:122] "RemoveContainer" containerID="d73e7e7a707c67d2a728ed8b3b126872a736435168aa8ef34a0b64c00d323a78"
Nov 02 20:45:03.532996 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.532975    5224 scope.go:122] "RemoveContainer" containerID="ad36a9eef5028113bedd79c456583fdc8ca01ff7a72937124c7297825d7179d5"
Nov 02 20:45:03.710514 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.710478    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:03.726832 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:03.726784    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj" podStartSLOduration=3.726753698 podStartE2EDuration="3.726753698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:03.724344742 +0000 UTC m=+3384.722033351" watchObservedRunningTime="2026-11-02 20:45:03.726753698 +0000 UTC m=+3384.724442306"
Nov 02 20:45:04.391581 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:04.391552    5224 scope.go:122] "RemoveContainer" containerID="5735b42745c43fd2ae11befdb54755bc5e16871876de14ed0034f217d7179bc6"
Nov 02 20:45:04.392052 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:04.391898    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt_odh-ai-gateway-infra(9bfed799-f1b8-4f33-97fb-a7d7d19313a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt" podUID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2"
Nov 02 20:45:04.393129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:04.393102    5224 generic.go:356] "Generic (PLEG): container finished" podID="2882985f-334d-4bf0-a48c-1fd5af4e70e2" containerID="3ae18dac3e41ccb519ed1b8d978a0014297e4f39c14c598b299ffc09fb0aec0b" exitCode=60
Nov 02 20:45:04.393220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:04.393180    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj" event={"ID":"2882985f-334d-4bf0-a48c-1fd5af4e70e2","Type":"ContainerDied","Data":"3ae18dac3e41ccb519ed1b8d978a0014297e4f39c14c598b299ffc09fb0aec0b"}
Nov 02 20:45:04.393271 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:04.393238    5224 scope.go:122] "RemoveContainer" containerID="f1e446a6895f12bf05855fb9acc92c808cf7ebb53ab3875212507aaec80987bf"
Nov 02 20:45:04.393403 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:04.393385    5224 scope.go:122] "RemoveContainer" containerID="3ae18dac3e41ccb519ed1b8d978a0014297e4f39c14c598b299ffc09fb0aec0b"
Nov 02 20:45:04.393652 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:04.393631    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj_odh-ai-gateway-infra(2882985f-334d-4bf0-a48c-1fd5af4e70e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj" podUID="2882985f-334d-4bf0-a48c-1fd5af4e70e2"
Nov 02 20:45:04.395157 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:04.395142    5224 scope.go:122] "RemoveContainer" containerID="a9939281b062a051ae1611b46ed51ca6c9b1bc2552763ecee7b50b8ddc780509"
Nov 02 20:45:04.395385 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:04.395363    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5_odh-ai-gateway-infra(31f0697b-2658-41aa-a282-31f018b79e86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5" podUID="31f0697b-2658-41aa-a282-31f018b79e86"
Nov 02 20:45:04.396653 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:04.396637    5224 scope.go:122] "RemoveContainer" containerID="d24b7ee1feedbfbc253253ec73f03a791cf75bb84177ed6dc0048d5fab389307"
Nov 02 20:45:04.396853 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:04.396835    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x_odh-ai-gateway-infra(8dca96fd-6559-47bf-bc64-f40571e6862c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x" podUID="8dca96fd-6559-47bf-bc64-f40571e6862c"
Nov 02 20:45:04.422209 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:04.422178    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:04.937154 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:04.937122    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:05.202447 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:05.202356    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:05.400105 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:05.400069    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:05.404452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:05.404428    5224 scope.go:122] "RemoveContainer" containerID="3ae18dac3e41ccb519ed1b8d978a0014297e4f39c14c598b299ffc09fb0aec0b"
Nov 02 20:45:05.404755 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:05.404732    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj_odh-ai-gateway-infra(2882985f-334d-4bf0-a48c-1fd5af4e70e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj" podUID="2882985f-334d-4bf0-a48c-1fd5af4e70e2"
Nov 02 20:45:09.937119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:09.937059    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:10.115992 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:10.115955    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:10.900221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:10.900181    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:14.954225 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:14.954182    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:15.160640 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:15.160611    5224 scope.go:122] "RemoveContainer" containerID="5735b42745c43fd2ae11befdb54755bc5e16871876de14ed0034f217d7179bc6"
Nov 02 20:45:15.160859 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:15.160667    5224 scope.go:122] "RemoveContainer" containerID="3ae18dac3e41ccb519ed1b8d978a0014297e4f39c14c598b299ffc09fb0aec0b"
Nov 02 20:45:15.471445 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:15.471411    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj" event={"ID":"2882985f-334d-4bf0-a48c-1fd5af4e70e2","Type":"ContainerStarted","Data":"4371b646a97eed55932edb41f41096693c665ead1c3b15f1d5549a93a8257754"}
Nov 02 20:45:15.635667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:15.635608    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:16.160816 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:16.160784    5224 scope.go:122] "RemoveContainer" containerID="a9939281b062a051ae1611b46ed51ca6c9b1bc2552763ecee7b50b8ddc780509"
Nov 02 20:45:16.480145 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:16.480114    5224 generic.go:356] "Generic (PLEG): container finished" podID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2" containerID="9847b86ba609d34b3b16b2459ae4617aa8a50dc3b626329948f359bb1c25023e" exitCode=60
Nov 02 20:45:16.480359 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:16.480195    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt" event={"ID":"9bfed799-f1b8-4f33-97fb-a7d7d19313a2","Type":"ContainerDied","Data":"9847b86ba609d34b3b16b2459ae4617aa8a50dc3b626329948f359bb1c25023e"}
Nov 02 20:45:16.480359 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:16.480268    5224 scope.go:122] "RemoveContainer" containerID="5735b42745c43fd2ae11befdb54755bc5e16871876de14ed0034f217d7179bc6"
Nov 02 20:45:16.480676 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:16.480656    5224 scope.go:122] "RemoveContainer" containerID="9847b86ba609d34b3b16b2459ae4617aa8a50dc3b626329948f359bb1c25023e"
Nov 02 20:45:16.481004 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:16.480984    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt_odh-ai-gateway-infra(9bfed799-f1b8-4f33-97fb-a7d7d19313a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt" podUID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2"
Nov 02 20:45:16.482153 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:16.482134    5224 generic.go:356] "Generic (PLEG): container finished" podID="2882985f-334d-4bf0-a48c-1fd5af4e70e2" containerID="4371b646a97eed55932edb41f41096693c665ead1c3b15f1d5549a93a8257754" exitCode=60
Nov 02 20:45:16.482252 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:16.482168    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj" event={"ID":"2882985f-334d-4bf0-a48c-1fd5af4e70e2","Type":"ContainerDied","Data":"4371b646a97eed55932edb41f41096693c665ead1c3b15f1d5549a93a8257754"}
Nov 02 20:45:16.482465 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:16.482453    5224 scope.go:122] "RemoveContainer" containerID="4371b646a97eed55932edb41f41096693c665ead1c3b15f1d5549a93a8257754"
Nov 02 20:45:16.482687 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:16.482673    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj_odh-ai-gateway-infra(2882985f-334d-4bf0-a48c-1fd5af4e70e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj" podUID="2882985f-334d-4bf0-a48c-1fd5af4e70e2"
Nov 02 20:45:16.492510 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:16.492491    5224 scope.go:122] "RemoveContainer" containerID="3ae18dac3e41ccb519ed1b8d978a0014297e4f39c14c598b299ffc09fb0aec0b"
Nov 02 20:45:16.602321 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:16.602239    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"]
Nov 02 20:45:17.161529 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.161493    5224 scope.go:122] "RemoveContainer" containerID="d24b7ee1feedbfbc253253ec73f03a791cf75bb84177ed6dc0048d5fab389307"
Nov 02 20:45:17.493179 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.493141    5224 generic.go:356] "Generic (PLEG): container finished" podID="31f0697b-2658-41aa-a282-31f018b79e86" containerID="eaffe22b414e3fe8335e948917a9664f22416de1384ed211bf80e0e0a767a694" exitCode=60
Nov 02 20:45:17.493371 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.493215    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5" event={"ID":"31f0697b-2658-41aa-a282-31f018b79e86","Type":"ContainerDied","Data":"eaffe22b414e3fe8335e948917a9664f22416de1384ed211bf80e0e0a767a694"}
Nov 02 20:45:17.493371 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.493272    5224 scope.go:122] "RemoveContainer" containerID="a9939281b062a051ae1611b46ed51ca6c9b1bc2552763ecee7b50b8ddc780509"
Nov 02 20:45:17.493694 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.493675    5224 scope.go:122] "RemoveContainer" containerID="eaffe22b414e3fe8335e948917a9664f22416de1384ed211bf80e0e0a767a694"
Nov 02 20:45:17.494039 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:17.494008    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5_odh-ai-gateway-infra(31f0697b-2658-41aa-a282-31f018b79e86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5" podUID="31f0697b-2658-41aa-a282-31f018b79e86"
Nov 02 20:45:17.495837 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.495813    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x" event={"ID":"8dca96fd-6559-47bf-bc64-f40571e6862c","Type":"ContainerStarted","Data":"7c9b8a43d9bd70cfe29acd4a135331b575798577cc6e8b77a77069b8096185ce"}
Nov 02 20:45:17.607539 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.607501    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"]
Nov 02 20:45:17.646332 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.646300    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"
Nov 02 20:45:17.714058 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.714024    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2882985f-334d-4bf0-a48c-1fd5af4e70e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2882985f-334d-4bf0-a48c-1fd5af4e70e2-maas-api-service-ca\") pod \"2882985f-334d-4bf0-a48c-1fd5af4e70e2\" (UID: \"2882985f-334d-4bf0-a48c-1fd5af4e70e2\") "
Nov 02 20:45:17.714262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.714143    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2882985f-334d-4bf0-a48c-1fd5af4e70e2-kube-api-access-qggxp\" (UniqueName: \"kubernetes.io/projected/2882985f-334d-4bf0-a48c-1fd5af4e70e2-kube-api-access-qggxp\") pod \"2882985f-334d-4bf0-a48c-1fd5af4e70e2\" (UID: \"2882985f-334d-4bf0-a48c-1fd5af4e70e2\") "
Nov 02 20:45:17.714480 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.714455    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2882985f-334d-4bf0-a48c-1fd5af4e70e2-maas-api-service-ca" pod "2882985f-334d-4bf0-a48c-1fd5af4e70e2" (UID: "2882985f-334d-4bf0-a48c-1fd5af4e70e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:45:17.716266 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.716243    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2882985f-334d-4bf0-a48c-1fd5af4e70e2-kube-api-access-qggxp" pod "2882985f-334d-4bf0-a48c-1fd5af4e70e2" (UID: "2882985f-334d-4bf0-a48c-1fd5af4e70e2"). InnerVolumeSpecName "kube-api-access-qggxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:45:17.776959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.776932    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"
Nov 02 20:45:17.815373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.815336    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qggxp\" (UniqueName: \"kubernetes.io/projected/2882985f-334d-4bf0-a48c-1fd5af4e70e2-kube-api-access-qggxp\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:45:17.815373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.815368    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2882985f-334d-4bf0-a48c-1fd5af4e70e2-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:45:17.838415 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.838368    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x" podStartSLOduration=17.838356129 podStartE2EDuration="17.838356129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:17.837169394 +0000 UTC m=+3398.834858003" watchObservedRunningTime="2026-11-02 20:45:17.838356129 +0000 UTC m=+3398.836044738"
Nov 02 20:45:17.916185 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.916090    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-maas-api-service-ca\") pod \"9bfed799-f1b8-4f33-97fb-a7d7d19313a2\" (UID: \"9bfed799-f1b8-4f33-97fb-a7d7d19313a2\") "
Nov 02 20:45:17.916350 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.916234    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-kube-api-access-cnbwc\" (UniqueName: \"kubernetes.io/projected/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-kube-api-access-cnbwc\") pod \"9bfed799-f1b8-4f33-97fb-a7d7d19313a2\" (UID: \"9bfed799-f1b8-4f33-97fb-a7d7d19313a2\") "
Nov 02 20:45:17.916478 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.916451    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-maas-api-service-ca" pod "9bfed799-f1b8-4f33-97fb-a7d7d19313a2" (UID: "9bfed799-f1b8-4f33-97fb-a7d7d19313a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:45:17.916605 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.916591    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:45:17.918508 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:17.918476    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-kube-api-access-cnbwc" pod "9bfed799-f1b8-4f33-97fb-a7d7d19313a2" (UID: "9bfed799-f1b8-4f33-97fb-a7d7d19313a2"). InnerVolumeSpecName "kube-api-access-cnbwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:45:18.017713 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.017675    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnbwc\" (UniqueName: \"kubernetes.io/projected/9bfed799-f1b8-4f33-97fb-a7d7d19313a2-kube-api-access-cnbwc\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:45:18.510124 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.510079    5224 generic.go:356] "Generic (PLEG): container finished" podID="8dca96fd-6559-47bf-bc64-f40571e6862c" containerID="7c9b8a43d9bd70cfe29acd4a135331b575798577cc6e8b77a77069b8096185ce" exitCode=60
Nov 02 20:45:18.510602 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.510143    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x" event={"ID":"8dca96fd-6559-47bf-bc64-f40571e6862c","Type":"ContainerDied","Data":"7c9b8a43d9bd70cfe29acd4a135331b575798577cc6e8b77a77069b8096185ce"}
Nov 02 20:45:18.510602 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.510215    5224 scope.go:122] "RemoveContainer" containerID="d24b7ee1feedbfbc253253ec73f03a791cf75bb84177ed6dc0048d5fab389307"
Nov 02 20:45:18.510739 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.510599    5224 scope.go:122] "RemoveContainer" containerID="7c9b8a43d9bd70cfe29acd4a135331b575798577cc6e8b77a77069b8096185ce"
Nov 02 20:45:18.510970 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:18.510944    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x_odh-ai-gateway-infra(8dca96fd-6559-47bf-bc64-f40571e6862c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x" podUID="8dca96fd-6559-47bf-bc64-f40571e6862c"
Nov 02 20:45:18.512251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.511973    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"
Nov 02 20:45:18.512251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.511970    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt" event={"ID":"9bfed799-f1b8-4f33-97fb-a7d7d19313a2","Type":"ContainerDied","Data":"8c73abcdac9e628559e3f13c4421ffcc51406a600c1f49dbb39515a7050c1ccc"}
Nov 02 20:45:18.513382 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.513359    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj" event={"ID":"2882985f-334d-4bf0-a48c-1fd5af4e70e2","Type":"ContainerDied","Data":"af4965827e97630cba61d77d6cfba4a1f74f232245bf28d344d7bc315c777e63"}
Nov 02 20:45:18.513382 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.513374    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"
Nov 02 20:45:18.523871 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.523849    5224 scope.go:122] "RemoveContainer" containerID="9847b86ba609d34b3b16b2459ae4617aa8a50dc3b626329948f359bb1c25023e"
Nov 02 20:45:18.533177 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.533155    5224 scope.go:122] "RemoveContainer" containerID="4371b646a97eed55932edb41f41096693c665ead1c3b15f1d5549a93a8257754"
Nov 02 20:45:18.618859 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.618805    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"]
Nov 02 20:45:18.744791 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.744727    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"]
Nov 02 20:45:18.752847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.752811    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894205-85kpt"]
Nov 02 20:45:18.784246 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.784217    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"
Nov 02 20:45:18.846759 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.846722    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"]
Nov 02 20:45:18.928898 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.928866    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f0697b-2658-41aa-a282-31f018b79e86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f0697b-2658-41aa-a282-31f018b79e86-maas-api-service-ca\") pod \"31f0697b-2658-41aa-a282-31f018b79e86\" (UID: \"31f0697b-2658-41aa-a282-31f018b79e86\") "
Nov 02 20:45:18.929082 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.928961    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f0697b-2658-41aa-a282-31f018b79e86-kube-api-access-mxs8h\" (UniqueName: \"kubernetes.io/projected/31f0697b-2658-41aa-a282-31f018b79e86-kube-api-access-mxs8h\") pod \"31f0697b-2658-41aa-a282-31f018b79e86\" (UID: \"31f0697b-2658-41aa-a282-31f018b79e86\") "
Nov 02 20:45:18.929319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.929294    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f0697b-2658-41aa-a282-31f018b79e86-maas-api-service-ca" pod "31f0697b-2658-41aa-a282-31f018b79e86" (UID: "31f0697b-2658-41aa-a282-31f018b79e86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:45:18.931108 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.931076    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f0697b-2658-41aa-a282-31f018b79e86-kube-api-access-mxs8h" pod "31f0697b-2658-41aa-a282-31f018b79e86" (UID: "31f0697b-2658-41aa-a282-31f018b79e86"). InnerVolumeSpecName "kube-api-access-mxs8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:45:18.976658 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.976621    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"]
Nov 02 20:45:18.980170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:18.980135    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894205-ggxzj"]
Nov 02 20:45:19.030104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.030017    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f0697b-2658-41aa-a282-31f018b79e86-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:45:19.030104 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.030042    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxs8h\" (UniqueName: \"kubernetes.io/projected/31f0697b-2658-41aa-a282-31f018b79e86-kube-api-access-mxs8h\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:45:19.168732 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.168699    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2882985f-334d-4bf0-a48c-1fd5af4e70e2" path="/var/lib/kubelet/pods/2882985f-334d-4bf0-a48c-1fd5af4e70e2/volumes"
Nov 02 20:45:19.169203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.169186    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2" path="/var/lib/kubelet/pods/9bfed799-f1b8-4f33-97fb-a7d7d19313a2/volumes"
Nov 02 20:45:19.521107 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.521076    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"
Nov 02 20:45:19.521612 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.521077    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5" event={"ID":"31f0697b-2658-41aa-a282-31f018b79e86","Type":"ContainerDied","Data":"0c93a7036abbdc020635a7e820794c631da126751cecc6d4b941cbcc644303d0"}
Nov 02 20:45:19.521612 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.521212    5224 scope.go:122] "RemoveContainer" containerID="eaffe22b414e3fe8335e948917a9664f22416de1384ed211bf80e0e0a767a694"
Nov 02 20:45:19.562665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.562618    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:19.677520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.677484    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"
Nov 02 20:45:19.689575 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.689542    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"]
Nov 02 20:45:19.700494 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.700462    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894205-bv9w5"]
Nov 02 20:45:19.840462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.840371    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dca96fd-6559-47bf-bc64-f40571e6862c-kube-api-access-pfjq4\" (UniqueName: \"kubernetes.io/projected/8dca96fd-6559-47bf-bc64-f40571e6862c-kube-api-access-pfjq4\") pod \"8dca96fd-6559-47bf-bc64-f40571e6862c\" (UID: \"8dca96fd-6559-47bf-bc64-f40571e6862c\") "
Nov 02 20:45:19.840462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.840429    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dca96fd-6559-47bf-bc64-f40571e6862c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dca96fd-6559-47bf-bc64-f40571e6862c-maas-api-service-ca\") pod \"8dca96fd-6559-47bf-bc64-f40571e6862c\" (UID: \"8dca96fd-6559-47bf-bc64-f40571e6862c\") "
Nov 02 20:45:19.840867 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.840843    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dca96fd-6559-47bf-bc64-f40571e6862c-maas-api-service-ca" pod "8dca96fd-6559-47bf-bc64-f40571e6862c" (UID: "8dca96fd-6559-47bf-bc64-f40571e6862c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:45:19.842570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.842547    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dca96fd-6559-47bf-bc64-f40571e6862c-kube-api-access-pfjq4" pod "8dca96fd-6559-47bf-bc64-f40571e6862c" (UID: "8dca96fd-6559-47bf-bc64-f40571e6862c"). InnerVolumeSpecName "kube-api-access-pfjq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:45:19.942228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.942190    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfjq4\" (UniqueName: \"kubernetes.io/projected/8dca96fd-6559-47bf-bc64-f40571e6862c-kube-api-access-pfjq4\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:45:19.942228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:19.942229    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dca96fd-6559-47bf-bc64-f40571e6862c-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:45:20.533949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:20.533913    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"
Nov 02 20:45:20.533949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:20.533930    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x" event={"ID":"8dca96fd-6559-47bf-bc64-f40571e6862c","Type":"ContainerDied","Data":"7d797b08949a08d15bc57cfcaa24809f26f73230a914c0e9ccd83deb3f7414d0"}
Nov 02 20:45:20.534468 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:20.534046    5224 scope.go:122] "RemoveContainer" containerID="7c9b8a43d9bd70cfe29acd4a135331b575798577cc6e8b77a77069b8096185ce"
Nov 02 20:45:20.701328 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:20.701292    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"]
Nov 02 20:45:20.706629 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:20.706593    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894205-76k6x"]
Nov 02 20:45:21.169745 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:21.169707    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f0697b-2658-41aa-a282-31f018b79e86" path="/var/lib/kubelet/pods/31f0697b-2658-41aa-a282-31f018b79e86/volumes"
Nov 02 20:45:21.170198 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:21.170183    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dca96fd-6559-47bf-bc64-f40571e6862c" path="/var/lib/kubelet/pods/8dca96fd-6559-47bf-bc64-f40571e6862c/volumes"
Nov 02 20:45:24.429140 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:24.429103    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:26.688617 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:26.688577    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:27.359696 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:27.359658    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:45:29.278761 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:29.278728    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:31.509011 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:31.508972    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:32.110510 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:32.110471    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:32.498674 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:32.498557    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:32.749167 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:32.749061    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:33.265155 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:33.265119    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:33.606264 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:33.606168    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:34.186574 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:34.186508    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:36.557142 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:36.557104    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:36.786714 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:36.786676    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:37.516199 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:37.516157    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:37.705559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:37.705521    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:37.887730 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:37.887636    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:39.470620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:39.470578    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:39.705672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:39.705633    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:39.916666 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:39.916600    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:44.217496 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:44.217461    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:44.433119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:44.433078    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:49.281482 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:49.281393    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:54.170000 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:54.169958    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:45:56.564644 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:56.564597    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:45:57.362551 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:45:57.362505    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:45:59.047617 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:45:59.047581    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:03.953971 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:03.953927    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:04.123822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:04.123788    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:04.763069 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:04.763035    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:08.850262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:08.850220    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:13.747578 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:13.747533    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:18.765484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:18.765439    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:23.644197 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:23.644152    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:27.359003 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:46:27.358965    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:46:28.602140 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:28.602096    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:30.136745 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:30.136700    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:31.075398 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:31.075347    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:31.961899 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:31.961860    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"]
Nov 02 20:46:32.017847 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:32.017814    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:33.462978 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:33.462939    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:38.362832 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:38.362751    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:43.244107 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:43.244065    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:46.984345 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:46.984301    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25" podUID="8eb533ba-79d6-45de-bb46-217781691c75" containerName="main" containerID="cri-o://85ca7be496e64efb1bdce33ecd9769516e9922dd782b091b03ba82f0c5420366" gracePeriod=45
Nov 02 20:46:47.252676 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.252612    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:46:47.326141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.326105    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb533ba-79d6-45de-bb46-217781691c75-kube-api-access-cgf5l\" (UniqueName: \"kubernetes.io/projected/8eb533ba-79d6-45de-bb46-217781691c75-kube-api-access-cgf5l\") pod \"8eb533ba-79d6-45de-bb46-217781691c75\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") "
Nov 02 20:46:47.326330 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.326186    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-model-cache\") pod \"8eb533ba-79d6-45de-bb46-217781691c75\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") "
Nov 02 20:46:47.326330 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.326241    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-dshm\") pod \"8eb533ba-79d6-45de-bb46-217781691c75\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") "
Nov 02 20:46:47.326330 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.326299    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-tmp-dir\") pod \"8eb533ba-79d6-45de-bb46-217781691c75\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") "
Nov 02 20:46:47.326330 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.326326    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb533ba-79d6-45de-bb46-217781691c75-tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb533ba-79d6-45de-bb46-217781691c75-tls-certs\") pod \"8eb533ba-79d6-45de-bb46-217781691c75\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") "
Nov 02 20:46:47.326559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.326349    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-home\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-home\") pod \"8eb533ba-79d6-45de-bb46-217781691c75\" (UID: \"8eb533ba-79d6-45de-bb46-217781691c75\") "
Nov 02 20:46:47.326559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.326503    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-model-cache" pod "8eb533ba-79d6-45de-bb46-217781691c75" (UID: "8eb533ba-79d6-45de-bb46-217781691c75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:46:47.326675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.326622    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-tmp-dir" pod "8eb533ba-79d6-45de-bb46-217781691c75" (UID: "8eb533ba-79d6-45de-bb46-217781691c75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:46:47.326721 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.326671    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-home" pod "8eb533ba-79d6-45de-bb46-217781691c75" (UID: "8eb533ba-79d6-45de-bb46-217781691c75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:46:47.326841 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.326823    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:46:47.326919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.326845    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:46:47.326919 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.326857    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:46:47.328559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.328533    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-dshm" pod "8eb533ba-79d6-45de-bb46-217781691c75" (UID: "8eb533ba-79d6-45de-bb46-217781691c75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:46:47.328667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.328564    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb533ba-79d6-45de-bb46-217781691c75-kube-api-access-cgf5l" pod "8eb533ba-79d6-45de-bb46-217781691c75" (UID: "8eb533ba-79d6-45de-bb46-217781691c75"). InnerVolumeSpecName "kube-api-access-cgf5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:46:47.328667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.328585    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb533ba-79d6-45de-bb46-217781691c75-tls-certs" pod "8eb533ba-79d6-45de-bb46-217781691c75" (UID: "8eb533ba-79d6-45de-bb46-217781691c75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:46:47.427533 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.427501    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb533ba-79d6-45de-bb46-217781691c75-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:46:47.427533 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.427529    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgf5l\" (UniqueName: \"kubernetes.io/projected/8eb533ba-79d6-45de-bb46-217781691c75-kube-api-access-cgf5l\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:46:47.427533 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.427540    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb533ba-79d6-45de-bb46-217781691c75-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:46:47.997529 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:47.997492    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:48.067191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:48.067153    5224 generic.go:356] "Generic (PLEG): container finished" podID="8eb533ba-79d6-45de-bb46-217781691c75" containerID="85ca7be496e64efb1bdce33ecd9769516e9922dd782b091b03ba82f0c5420366" exitCode=0
Nov 02 20:46:48.067373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:48.067209    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"
Nov 02 20:46:48.067373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:48.067241    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25" event={"ID":"8eb533ba-79d6-45de-bb46-217781691c75","Type":"ContainerDied","Data":"85ca7be496e64efb1bdce33ecd9769516e9922dd782b091b03ba82f0c5420366"}
Nov 02 20:46:48.067373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:48.067288    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25" event={"ID":"8eb533ba-79d6-45de-bb46-217781691c75","Type":"ContainerDied","Data":"28d6512dcca4e4873872cd15c3ff7060ea06a65409b6a7deb79fef2ad0fb6c55"}
Nov 02 20:46:48.067373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:48.067306    5224 scope.go:122] "RemoveContainer" containerID="85ca7be496e64efb1bdce33ecd9769516e9922dd782b091b03ba82f0c5420366"
Nov 02 20:46:48.077890 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:48.077868    5224 scope.go:122] "RemoveContainer" containerID="85ca7be496e64efb1bdce33ecd9769516e9922dd782b091b03ba82f0c5420366"
Nov 02 20:46:48.078172 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:46:48.078152    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ca7be496e64efb1bdce33ecd9769516e9922dd782b091b03ba82f0c5420366\": container with ID starting with 85ca7be496e64efb1bdce33ecd9769516e9922dd782b091b03ba82f0c5420366 not found: ID does not exist" containerID="85ca7be496e64efb1bdce33ecd9769516e9922dd782b091b03ba82f0c5420366"
Nov 02 20:46:48.078215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:48.078186    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ca7be496e64efb1bdce33ecd9769516e9922dd782b091b03ba82f0c5420366"} err="failed to get container status \"85ca7be496e64efb1bdce33ecd9769516e9922dd782b091b03ba82f0c5420366\": rpc error: code = NotFound desc = could not find container \"85ca7be496e64efb1bdce33ecd9769516e9922dd782b091b03ba82f0c5420366\": container with ID starting with 85ca7be496e64efb1bdce33ecd9769516e9922dd782b091b03ba82f0c5420366 not found: ID does not exist"
Nov 02 20:46:48.169335 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:48.169298    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:48.233876 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:48.233843    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"]
Nov 02 20:46:48.237499 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:48.237476    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/auth-test-model-76c687-kserve-55f896d9cc-2rv25"]
Nov 02 20:46:48.478611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:48.478576    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:49.168715 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:49.168678    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb533ba-79d6-45de-bb46-217781691c75" path="/var/lib/kubelet/pods/8eb533ba-79d6-45de-bb46-217781691c75/volumes"
Nov 02 20:46:52.887470 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:52.887418    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:53.110204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:53.110149    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:46:57.359291 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:46:57.359255    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:46:57.969921 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:46:57.969883    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:00.146706 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:00.146670    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"]
Nov 02 20:47:00.202098 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:00.202058    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:01.059034 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:01.058998    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"]
Nov 02 20:47:01.102632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:01.102596    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:01.677845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:01.677805    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:01.829010 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:01.828977    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:47:01.852377 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:01.852341    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:02.224787 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:02.224731    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:03.488160 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:03.488121    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:03.667243 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:03.667203    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:04.387011 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:04.386972    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:05.283579 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:05.283314    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:06.321462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:06.321423    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:06.552599 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:06.552566    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:07.123601 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:07.123560    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:07.313563 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:07.313526    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:07.499711 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:07.499626    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:08.269891 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:08.269855    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:09.020326 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:09.020294    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:09.229058 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:09.229023    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:09.396050 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:09.395960    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:11.542667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:11.542628    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:11.757993 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:11.757952    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:13.673833 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:13.673796    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:13.941041 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:13.940964    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:14.111051 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.111013    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:14.308936 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.308901    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"]
Nov 02 20:47:14.309353 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309341    5224 cpu_manager.go:414] "containerMap: removing container" podUID="8dca96fd-6559-47bf-bc64-f40571e6862c" containerName="cleanup"
Nov 02 20:47:14.309404 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309355    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dca96fd-6559-47bf-bc64-f40571e6862c" containerName="cleanup"
Nov 02 20:47:14.309404 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309369    5224 cpu_manager.go:414] "containerMap: removing container" podUID="2882985f-334d-4bf0-a48c-1fd5af4e70e2" containerName="cleanup"
Nov 02 20:47:14.309404 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309375    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2882985f-334d-4bf0-a48c-1fd5af4e70e2" containerName="cleanup"
Nov 02 20:47:14.309404 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309382    5224 cpu_manager.go:414] "containerMap: removing container" podUID="8dca96fd-6559-47bf-bc64-f40571e6862c" containerName="cleanup"
Nov 02 20:47:14.309404 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309387    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dca96fd-6559-47bf-bc64-f40571e6862c" containerName="cleanup"
Nov 02 20:47:14.309404 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309400    5224 cpu_manager.go:414] "containerMap: removing container" podUID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2" containerName="cleanup"
Nov 02 20:47:14.309404 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309404    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309414    5224 cpu_manager.go:414] "containerMap: removing container" podUID="31f0697b-2658-41aa-a282-31f018b79e86" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309419    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f0697b-2658-41aa-a282-31f018b79e86" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309429    5224 cpu_manager.go:414] "containerMap: removing container" podUID="2882985f-334d-4bf0-a48c-1fd5af4e70e2" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309433    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2882985f-334d-4bf0-a48c-1fd5af4e70e2" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309441    5224 cpu_manager.go:414] "containerMap: removing container" podUID="8dca96fd-6559-47bf-bc64-f40571e6862c" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309445    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dca96fd-6559-47bf-bc64-f40571e6862c" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309451    5224 cpu_manager.go:414] "containerMap: removing container" podUID="31f0697b-2658-41aa-a282-31f018b79e86" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309455    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f0697b-2658-41aa-a282-31f018b79e86" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309471    5224 cpu_manager.go:414] "containerMap: removing container" podUID="8eb533ba-79d6-45de-bb46-217781691c75" containerName="main"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309476    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb533ba-79d6-45de-bb46-217781691c75" containerName="main"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309484    5224 cpu_manager.go:414] "containerMap: removing container" podUID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309489    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309501    5224 cpu_manager.go:414] "containerMap: removing container" podUID="31f0697b-2658-41aa-a282-31f018b79e86" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309505    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f0697b-2658-41aa-a282-31f018b79e86" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309572    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f0697b-2658-41aa-a282-31f018b79e86" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309579    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309588    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb533ba-79d6-45de-bb46-217781691c75" containerName="main"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309594    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="2882985f-334d-4bf0-a48c-1fd5af4e70e2" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309600    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309607    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dca96fd-6559-47bf-bc64-f40571e6862c" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309613    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="2882985f-334d-4bf0-a48c-1fd5af4e70e2" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309618    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dca96fd-6559-47bf-bc64-f40571e6862c" containerName="cleanup"
Nov 02 20:47:14.309622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.309624    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f0697b-2658-41aa-a282-31f018b79e86" containerName="cleanup"
Nov 02 20:47:14.312845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.312829    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"
Nov 02 20:47:14.335191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.335162    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb55d\" (UniqueName: \"kubernetes.io/projected/cd6be353-2b67-4544-89fd-96c6233ac47e-kube-api-access-kb55d\") pod \"payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh\" (UID: \"cd6be353-2b67-4544-89fd-96c6233ac47e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"
Nov 02 20:47:14.335328 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.335231    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6be353-2b67-4544-89fd-96c6233ac47e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh\" (UID: \"cd6be353-2b67-4544-89fd-96c6233ac47e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"
Nov 02 20:47:14.368884 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.368092    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"]
Nov 02 20:47:14.416048 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.416017    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-608c11\""
Nov 02 20:47:14.416048 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.416038    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-608c11-dockercfg-jchzt\""
Nov 02 20:47:14.436476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.436443    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb55d\" (UniqueName: \"kubernetes.io/projected/cd6be353-2b67-4544-89fd-96c6233ac47e-kube-api-access-kb55d\") pod \"payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh\" (UID: \"cd6be353-2b67-4544-89fd-96c6233ac47e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"
Nov 02 20:47:14.436685 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.436542    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6be353-2b67-4544-89fd-96c6233ac47e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh\" (UID: \"cd6be353-2b67-4544-89fd-96c6233ac47e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"
Nov 02 20:47:14.437274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.437255    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6be353-2b67-4544-89fd-96c6233ac47e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh\" (UID: \"cd6be353-2b67-4544-89fd-96c6233ac47e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"
Nov 02 20:47:14.469732 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.469690    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"]
Nov 02 20:47:14.470250 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.470231    5224 cpu_manager.go:414] "containerMap: removing container" podUID="2882985f-334d-4bf0-a48c-1fd5af4e70e2" containerName="cleanup"
Nov 02 20:47:14.470351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.470252    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2882985f-334d-4bf0-a48c-1fd5af4e70e2" containerName="cleanup"
Nov 02 20:47:14.470351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.470278    5224 cpu_manager.go:414] "containerMap: removing container" podUID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2" containerName="cleanup"
Nov 02 20:47:14.470351 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.470285    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2" containerName="cleanup"
Nov 02 20:47:14.470518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.470450    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f0697b-2658-41aa-a282-31f018b79e86" containerName="cleanup"
Nov 02 20:47:14.470518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.470471    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfed799-f1b8-4f33-97fb-a7d7d19313a2" containerName="cleanup"
Nov 02 20:47:14.470518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.470485    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="2882985f-334d-4bf0-a48c-1fd5af4e70e2" containerName="cleanup"
Nov 02 20:47:14.470518 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.470495    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dca96fd-6559-47bf-bc64-f40571e6862c" containerName="cleanup"
Nov 02 20:47:14.474573 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.474549    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"
Nov 02 20:47:14.494320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.494285    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb55d\" (UniqueName: \"kubernetes.io/projected/cd6be353-2b67-4544-89fd-96c6233ac47e-kube-api-access-kb55d\") pod \"payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh\" (UID: \"cd6be353-2b67-4544-89fd-96c6233ac47e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"
Nov 02 20:47:14.525100 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.525066    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"]
Nov 02 20:47:14.538269 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.538236    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj\" (UID: \"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"
Nov 02 20:47:14.538451 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.538309    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgxjj\" (UniqueName: \"kubernetes.io/projected/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-kube-api-access-fgxjj\") pod \"payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj\" (UID: \"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"
Nov 02 20:47:14.626109 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.626021    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"
Nov 02 20:47:14.639813 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.639755    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj\" (UID: \"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"
Nov 02 20:47:14.639942 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.639902    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgxjj\" (UniqueName: \"kubernetes.io/projected/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-kube-api-access-fgxjj\") pod \"payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj\" (UID: \"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"
Nov 02 20:47:14.640439 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.640421    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj\" (UID: \"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"
Nov 02 20:47:14.698866 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.698830    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgxjj\" (UniqueName: \"kubernetes.io/projected/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-kube-api-access-fgxjj\") pod \"payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj\" (UID: \"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"
Nov 02 20:47:14.787961 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:14.787927    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"
Nov 02 20:47:14.997260 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:47:14.997229    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6be353_2b67_4544_89fd_96c6233ac47e.slice/crio-016f61cc00acb7c2fb1a4ad4ea500457776aa2d406f7aab9849f16eda3e3df02 WatchSource:0}: Error finding container 016f61cc00acb7c2fb1a4ad4ea500457776aa2d406f7aab9849f16eda3e3df02: Status 404 returned error can't find the container with id 016f61cc00acb7c2fb1a4ad4ea500457776aa2d406f7aab9849f16eda3e3df02
Nov 02 20:47:15.000365 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.000330    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"]
Nov 02 20:47:15.159501 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:47:15.159472    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d4b8d6_1ca9_4eb0_ba0d_3a5ba813c267.slice/crio-552998f838a8cc827580a1803c8ce23e72d5ef788f0a81b8180dfb9c500a63c9 WatchSource:0}: Error finding container 552998f838a8cc827580a1803c8ce23e72d5ef788f0a81b8180dfb9c500a63c9: Status 404 returned error can't find the container with id 552998f838a8cc827580a1803c8ce23e72d5ef788f0a81b8180dfb9c500a63c9
Nov 02 20:47:15.167018 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.166990    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs" podUID="34024b80-0661-4410-b654-1eaf65fe14fd" containerName="main" containerID="cri-o://eeb2c35cc170f840a6b2fb2ebee29f4f7886338e6d1eded08207dd5efff232f5" gracePeriod=45
Nov 02 20:47:15.169108 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.169086    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"]
Nov 02 20:47:15.235845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.235810    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh" event={"ID":"cd6be353-2b67-4544-89fd-96c6233ac47e","Type":"ContainerStarted","Data":"3b9b4965a37d5be5c52834ec213e0a8d7ec57de4416d5b802060db26b0b60669"}
Nov 02 20:47:15.236051 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.235877    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh" event={"ID":"cd6be353-2b67-4544-89fd-96c6233ac47e","Type":"ContainerStarted","Data":"016f61cc00acb7c2fb1a4ad4ea500457776aa2d406f7aab9849f16eda3e3df02"}
Nov 02 20:47:15.236051 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.236025    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"
Nov 02 20:47:15.237203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.237177    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj" event={"ID":"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267","Type":"ContainerStarted","Data":"552998f838a8cc827580a1803c8ce23e72d5ef788f0a81b8180dfb9c500a63c9"}
Nov 02 20:47:15.353291 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.353241    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh" podStartSLOduration=1.353226721 podStartE2EDuration="1.353226721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:15.351376703 +0000 UTC m=+3516.349065311" watchObservedRunningTime="2026-11-02 20:47:15.353226721 +0000 UTC m=+3516.350915329"
Nov 02 20:47:15.414353 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.414324    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:47:15.451141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.451100    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-home\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-home\") pod \"34024b80-0661-4410-b654-1eaf65fe14fd\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") "
Nov 02 20:47:15.451319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.451155    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34024b80-0661-4410-b654-1eaf65fe14fd-kube-api-access-qjb7b\" (UniqueName: \"kubernetes.io/projected/34024b80-0661-4410-b654-1eaf65fe14fd-kube-api-access-qjb7b\") pod \"34024b80-0661-4410-b654-1eaf65fe14fd\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") "
Nov 02 20:47:15.451319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.451187    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-model-cache\") pod \"34024b80-0661-4410-b654-1eaf65fe14fd\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") "
Nov 02 20:47:15.451319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.451234    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-tmp-dir\") pod \"34024b80-0661-4410-b654-1eaf65fe14fd\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") "
Nov 02 20:47:15.451319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.451296    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-dshm\") pod \"34024b80-0661-4410-b654-1eaf65fe14fd\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") "
Nov 02 20:47:15.451551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.451338    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34024b80-0661-4410-b654-1eaf65fe14fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/34024b80-0661-4410-b654-1eaf65fe14fd-tls-certs\") pod \"34024b80-0661-4410-b654-1eaf65fe14fd\" (UID: \"34024b80-0661-4410-b654-1eaf65fe14fd\") "
Nov 02 20:47:15.451551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.451409    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-home" pod "34024b80-0661-4410-b654-1eaf65fe14fd" (UID: "34024b80-0661-4410-b654-1eaf65fe14fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:47:15.451551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.451471    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-model-cache" pod "34024b80-0661-4410-b654-1eaf65fe14fd" (UID: "34024b80-0661-4410-b654-1eaf65fe14fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:47:15.451551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.451515    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-tmp-dir" pod "34024b80-0661-4410-b654-1eaf65fe14fd" (UID: "34024b80-0661-4410-b654-1eaf65fe14fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:47:15.452270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.452240    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:15.452270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.452272    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:15.452481 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.452287    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:15.457811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.456555    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34024b80-0661-4410-b654-1eaf65fe14fd-tls-certs" pod "34024b80-0661-4410-b654-1eaf65fe14fd" (UID: "34024b80-0661-4410-b654-1eaf65fe14fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:47:15.463592 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.463556    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34024b80-0661-4410-b654-1eaf65fe14fd-kube-api-access-qjb7b" pod "34024b80-0661-4410-b654-1eaf65fe14fd" (UID: "34024b80-0661-4410-b654-1eaf65fe14fd"). InnerVolumeSpecName "kube-api-access-qjb7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:47:15.463751 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.463690    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-dshm" pod "34024b80-0661-4410-b654-1eaf65fe14fd" (UID: "34024b80-0661-4410-b654-1eaf65fe14fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:47:15.553160 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.553058    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34024b80-0661-4410-b654-1eaf65fe14fd-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:15.553160 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.553097    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34024b80-0661-4410-b654-1eaf65fe14fd-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:15.553160 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:15.553114    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjb7b\" (UniqueName: \"kubernetes.io/projected/34024b80-0661-4410-b654-1eaf65fe14fd-kube-api-access-qjb7b\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:16.079737 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.079690    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn" podUID="102d743c-71ea-4b15-b624-dca42b2882f9" containerName="main" containerID="cri-o://761291d8dcb5ae5b2e823da42247ff00613c1d7b902361492299507deffc0ff0" gracePeriod=45
Nov 02 20:47:16.246220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.246184    5224 generic.go:356] "Generic (PLEG): container finished" podID="102d743c-71ea-4b15-b624-dca42b2882f9" containerID="761291d8dcb5ae5b2e823da42247ff00613c1d7b902361492299507deffc0ff0" exitCode=0
Nov 02 20:47:16.246406 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.246266    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn" event={"ID":"102d743c-71ea-4b15-b624-dca42b2882f9","Type":"ContainerDied","Data":"761291d8dcb5ae5b2e823da42247ff00613c1d7b902361492299507deffc0ff0"}
Nov 02 20:47:16.248190 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.248157    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj" event={"ID":"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267","Type":"ContainerStarted","Data":"6da135b40f664c0b1f62892c72ac770c4753af694aae691825daf7f5d2fd71b3"}
Nov 02 20:47:16.248373 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.248356    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"
Nov 02 20:47:16.249638 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.249610    5224 generic.go:356] "Generic (PLEG): container finished" podID="34024b80-0661-4410-b654-1eaf65fe14fd" containerID="eeb2c35cc170f840a6b2fb2ebee29f4f7886338e6d1eded08207dd5efff232f5" exitCode=0
Nov 02 20:47:16.249757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.249659    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"
Nov 02 20:47:16.249757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.249676    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs" event={"ID":"34024b80-0661-4410-b654-1eaf65fe14fd","Type":"ContainerDied","Data":"eeb2c35cc170f840a6b2fb2ebee29f4f7886338e6d1eded08207dd5efff232f5"}
Nov 02 20:47:16.249757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.249721    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs" event={"ID":"34024b80-0661-4410-b654-1eaf65fe14fd","Type":"ContainerDied","Data":"8349024e625f48872cba882f74fc313cd605f5df5927b18d954d860615c62f98"}
Nov 02 20:47:16.249757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.249750    5224 scope.go:122] "RemoveContainer" containerID="eeb2c35cc170f840a6b2fb2ebee29f4f7886338e6d1eded08207dd5efff232f5"
Nov 02 20:47:16.263971 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.262475    5224 scope.go:122] "RemoveContainer" containerID="eeb2c35cc170f840a6b2fb2ebee29f4f7886338e6d1eded08207dd5efff232f5"
Nov 02 20:47:16.263971 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:47:16.262810    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb2c35cc170f840a6b2fb2ebee29f4f7886338e6d1eded08207dd5efff232f5\": container with ID starting with eeb2c35cc170f840a6b2fb2ebee29f4f7886338e6d1eded08207dd5efff232f5 not found: ID does not exist" containerID="eeb2c35cc170f840a6b2fb2ebee29f4f7886338e6d1eded08207dd5efff232f5"
Nov 02 20:47:16.263971 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.262846    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb2c35cc170f840a6b2fb2ebee29f4f7886338e6d1eded08207dd5efff232f5"} err="failed to get container status \"eeb2c35cc170f840a6b2fb2ebee29f4f7886338e6d1eded08207dd5efff232f5\": rpc error: code = NotFound desc = could not find container \"eeb2c35cc170f840a6b2fb2ebee29f4f7886338e6d1eded08207dd5efff232f5\": container with ID starting with eeb2c35cc170f840a6b2fb2ebee29f4f7886338e6d1eded08207dd5efff232f5 not found: ID does not exist"
Nov 02 20:47:16.357596 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.357567    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:47:16.364734 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.364688    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj" podStartSLOduration=2.364675028 podStartE2EDuration="2.364675028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:16.364006562 +0000 UTC m=+3517.361695172" watchObservedRunningTime="2026-11-02 20:47:16.364675028 +0000 UTC m=+3517.362363636"
Nov 02 20:47:16.400188 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.400147    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs" podUID="34024b80-0661-4410-b654-1eaf65fe14fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.150:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:47:16.463084 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.463049    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-home\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-home\") pod \"102d743c-71ea-4b15-b624-dca42b2882f9\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") "
Nov 02 20:47:16.463268 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.463113    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/102d743c-71ea-4b15-b624-dca42b2882f9-kube-api-access-682rf\" (UniqueName: \"kubernetes.io/projected/102d743c-71ea-4b15-b624-dca42b2882f9-kube-api-access-682rf\") pod \"102d743c-71ea-4b15-b624-dca42b2882f9\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") "
Nov 02 20:47:16.463268 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.463148    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-dshm\") pod \"102d743c-71ea-4b15-b624-dca42b2882f9\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") "
Nov 02 20:47:16.463268 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.463182    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-model-cache\") pod \"102d743c-71ea-4b15-b624-dca42b2882f9\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") "
Nov 02 20:47:16.463268 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.463235    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/102d743c-71ea-4b15-b624-dca42b2882f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/102d743c-71ea-4b15-b624-dca42b2882f9-tls-certs\") pod \"102d743c-71ea-4b15-b624-dca42b2882f9\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") "
Nov 02 20:47:16.463268 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.463262    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-tmp-dir\") pod \"102d743c-71ea-4b15-b624-dca42b2882f9\" (UID: \"102d743c-71ea-4b15-b624-dca42b2882f9\") "
Nov 02 20:47:16.463474 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.463413    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-home" pod "102d743c-71ea-4b15-b624-dca42b2882f9" (UID: "102d743c-71ea-4b15-b624-dca42b2882f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:47:16.463474 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.463425    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-model-cache" pod "102d743c-71ea-4b15-b624-dca42b2882f9" (UID: "102d743c-71ea-4b15-b624-dca42b2882f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:47:16.463630 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.463611    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-tmp-dir" pod "102d743c-71ea-4b15-b624-dca42b2882f9" (UID: "102d743c-71ea-4b15-b624-dca42b2882f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:47:16.463686 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.463672    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:16.463742 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.463692    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:16.465584 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.465558    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-dshm" pod "102d743c-71ea-4b15-b624-dca42b2882f9" (UID: "102d743c-71ea-4b15-b624-dca42b2882f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:47:16.465699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.465581    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102d743c-71ea-4b15-b624-dca42b2882f9-tls-certs" pod "102d743c-71ea-4b15-b624-dca42b2882f9" (UID: "102d743c-71ea-4b15-b624-dca42b2882f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:47:16.465699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.465630    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102d743c-71ea-4b15-b624-dca42b2882f9-kube-api-access-682rf" pod "102d743c-71ea-4b15-b624-dca42b2882f9" (UID: "102d743c-71ea-4b15-b624-dca42b2882f9"). InnerVolumeSpecName "kube-api-access-682rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:47:16.564338 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.564301    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102d743c-71ea-4b15-b624-dca42b2882f9-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:16.564338 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.564328    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:16.564338 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.564338    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-682rf\" (UniqueName: \"kubernetes.io/projected/102d743c-71ea-4b15-b624-dca42b2882f9-kube-api-access-682rf\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:16.564338 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.564348    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102d743c-71ea-4b15-b624-dca42b2882f9-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:16.598599 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.598525    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"]
Nov 02 20:47:16.601757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:16.601720    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/auth-test-model-6b12ae-kserve-79fc68b74f-vmjfs"]
Nov 02 20:47:17.167228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:17.167196    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34024b80-0661-4410-b654-1eaf65fe14fd" path="/var/lib/kubelet/pods/34024b80-0661-4410-b654-1eaf65fe14fd/volumes"
Nov 02 20:47:17.256950 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:17.256925    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"
Nov 02 20:47:17.257141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:17.256957    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn" event={"ID":"102d743c-71ea-4b15-b624-dca42b2882f9","Type":"ContainerDied","Data":"b0b49a5828916e6995620c39c57130f1df7a52b14585f65d8b982341726e0bef"}
Nov 02 20:47:17.257141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:17.257007    5224 scope.go:122] "RemoveContainer" containerID="761291d8dcb5ae5b2e823da42247ff00613c1d7b902361492299507deffc0ff0"
Nov 02 20:47:17.424248 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:17.424151    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"]
Nov 02 20:47:17.427389 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:17.427362    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-578636/auth-test-model-578636-kserve-67f69f6bbf-k8ffn"]
Nov 02 20:47:18.504848 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.504745    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:18.735515 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.735475    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:18.845735 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.845700    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"]
Nov 02 20:47:18.846281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.846260    5224 cpu_manager.go:414] "containerMap: removing container" podUID="102d743c-71ea-4b15-b624-dca42b2882f9" containerName="main"
Nov 02 20:47:18.846281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.846282    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102d743c-71ea-4b15-b624-dca42b2882f9" containerName="main"
Nov 02 20:47:18.846464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.846308    5224 cpu_manager.go:414] "containerMap: removing container" podUID="34024b80-0661-4410-b654-1eaf65fe14fd" containerName="main"
Nov 02 20:47:18.846464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.846315    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34024b80-0661-4410-b654-1eaf65fe14fd" containerName="main"
Nov 02 20:47:18.846573 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.846464    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="102d743c-71ea-4b15-b624-dca42b2882f9" containerName="main"
Nov 02 20:47:18.846573 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.846479    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="34024b80-0661-4410-b654-1eaf65fe14fd" containerName="main"
Nov 02 20:47:18.851220 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.851197    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"
Nov 02 20:47:18.887636 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.887611    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwchq\" (UniqueName: \"kubernetes.io/projected/01316187-77b6-4788-b77f-72b8482f42a0-kube-api-access-qwchq\") pod \"payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s\" (UID: \"01316187-77b6-4788-b77f-72b8482f42a0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"
Nov 02 20:47:18.887757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.887734    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01316187-77b6-4788-b77f-72b8482f42a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s\" (UID: \"01316187-77b6-4788-b77f-72b8482f42a0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"
Nov 02 20:47:18.901206 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.901182    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"]
Nov 02 20:47:18.950125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.950100    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-720bda\""
Nov 02 20:47:18.950125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.950108    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-720bda-dockercfg-qf665\""
Nov 02 20:47:18.979130 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.979091    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:18.989054 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.989026    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwchq\" (UniqueName: \"kubernetes.io/projected/01316187-77b6-4788-b77f-72b8482f42a0-kube-api-access-qwchq\") pod \"payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s\" (UID: \"01316187-77b6-4788-b77f-72b8482f42a0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"
Nov 02 20:47:18.989201 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.989182    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01316187-77b6-4788-b77f-72b8482f42a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s\" (UID: \"01316187-77b6-4788-b77f-72b8482f42a0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"
Nov 02 20:47:18.989980 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:18.989955    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01316187-77b6-4788-b77f-72b8482f42a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s\" (UID: \"01316187-77b6-4788-b77f-72b8482f42a0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"
Nov 02 20:47:19.016789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.014024    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"]
Nov 02 20:47:19.021786 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.020004    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"
Nov 02 20:47:19.047710 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.047679    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwchq\" (UniqueName: \"kubernetes.io/projected/01316187-77b6-4788-b77f-72b8482f42a0-kube-api-access-qwchq\") pod \"payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s\" (UID: \"01316187-77b6-4788-b77f-72b8482f42a0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"
Nov 02 20:47:19.071071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.071040    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"]
Nov 02 20:47:19.090544 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.090508    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h7dq\" (UniqueName: \"kubernetes.io/projected/4b6258ad-7932-485c-aece-f41b6fda4c6e-kube-api-access-4h7dq\") pod \"payload-processing-e2e-autores-720bda-765b646cfc-9954t\" (UID: \"4b6258ad-7932-485c-aece-f41b6fda4c6e\") " pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"
Nov 02 20:47:19.090708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.090685    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6258ad-7932-485c-aece-f41b6fda4c6e-plugins-config-volume\") pod \"payload-processing-e2e-autores-720bda-765b646cfc-9954t\" (UID: \"4b6258ad-7932-485c-aece-f41b6fda4c6e\") " pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"
Nov 02 20:47:19.164531 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.164470    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"
Nov 02 20:47:19.168907 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.168878    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="102d743c-71ea-4b15-b624-dca42b2882f9" path="/var/lib/kubelet/pods/102d743c-71ea-4b15-b624-dca42b2882f9/volumes"
Nov 02 20:47:19.191630 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.191606    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h7dq\" (UniqueName: \"kubernetes.io/projected/4b6258ad-7932-485c-aece-f41b6fda4c6e-kube-api-access-4h7dq\") pod \"payload-processing-e2e-autores-720bda-765b646cfc-9954t\" (UID: \"4b6258ad-7932-485c-aece-f41b6fda4c6e\") " pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"
Nov 02 20:47:19.191739 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.191675    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6258ad-7932-485c-aece-f41b6fda4c6e-plugins-config-volume\") pod \"payload-processing-e2e-autores-720bda-765b646cfc-9954t\" (UID: \"4b6258ad-7932-485c-aece-f41b6fda4c6e\") " pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"
Nov 02 20:47:19.192355 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.192335    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6258ad-7932-485c-aece-f41b6fda4c6e-plugins-config-volume\") pod \"payload-processing-e2e-autores-720bda-765b646cfc-9954t\" (UID: \"4b6258ad-7932-485c-aece-f41b6fda4c6e\") " pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"
Nov 02 20:47:19.249418 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.249384    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h7dq\" (UniqueName: \"kubernetes.io/projected/4b6258ad-7932-485c-aece-f41b6fda4c6e-kube-api-access-4h7dq\") pod \"payload-processing-e2e-autores-720bda-765b646cfc-9954t\" (UID: \"4b6258ad-7932-485c-aece-f41b6fda4c6e\") " pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"
Nov 02 20:47:19.348854 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.348819    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"
Nov 02 20:47:19.531683 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:47:19.531652    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01316187_77b6_4788_b77f_72b8482f42a0.slice/crio-8c81e7adeae4bea0267fdcde2dfe338fc9720f0a5d619e3c2b686df881e1560f WatchSource:0}: Error finding container 8c81e7adeae4bea0267fdcde2dfe338fc9720f0a5d619e3c2b686df881e1560f: Status 404 returned error can't find the container with id 8c81e7adeae4bea0267fdcde2dfe338fc9720f0a5d619e3c2b686df881e1560f
Nov 02 20:47:19.534880 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.534855    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"]
Nov 02 20:47:19.716475 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:47:19.716447    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6258ad_7932_485c_aece_f41b6fda4c6e.slice/crio-501c9da4c28cdf5446b28f8f28f9911f14b76cb12c4efdb85c1c39837ae8b02a WatchSource:0}: Error finding container 501c9da4c28cdf5446b28f8f28f9911f14b76cb12c4efdb85c1c39837ae8b02a: Status 404 returned error can't find the container with id 501c9da4c28cdf5446b28f8f28f9911f14b76cb12c4efdb85c1c39837ae8b02a
Nov 02 20:47:19.719942 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:19.719916    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"]
Nov 02 20:47:20.277418 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:20.277368    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t" event={"ID":"4b6258ad-7932-485c-aece-f41b6fda4c6e","Type":"ContainerStarted","Data":"9876ce01baf720beb90286d53334eab30db3db517aafea3457eb1970cd56afdb"}
Nov 02 20:47:20.277623 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:20.277436    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t" event={"ID":"4b6258ad-7932-485c-aece-f41b6fda4c6e","Type":"ContainerStarted","Data":"501c9da4c28cdf5446b28f8f28f9911f14b76cb12c4efdb85c1c39837ae8b02a"}
Nov 02 20:47:20.277623 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:20.277466    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"
Nov 02 20:47:20.278983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:20.278957    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s" event={"ID":"01316187-77b6-4788-b77f-72b8482f42a0","Type":"ContainerStarted","Data":"f20aafdea0cad06afd68ecaf3d45b44e41e2c6c20b513ade5b6694c65e7a6c33"}
Nov 02 20:47:20.279066 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:20.278999    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s" event={"ID":"01316187-77b6-4788-b77f-72b8482f42a0","Type":"ContainerStarted","Data":"8c81e7adeae4bea0267fdcde2dfe338fc9720f0a5d619e3c2b686df881e1560f"}
Nov 02 20:47:20.279066 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:20.279021    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"
Nov 02 20:47:20.393328 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:20.393277    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t" podStartSLOduration=2.393262816 podStartE2EDuration="2.393262816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:20.392357469 +0000 UTC m=+3521.390046090" watchObservedRunningTime="2026-11-02 20:47:20.393262816 +0000 UTC m=+3521.390951425"
Nov 02 20:47:20.510740 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:20.510676    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s" podStartSLOduration=2.510659605 podStartE2EDuration="2.510659605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:20.507182303 +0000 UTC m=+3521.504870909" watchObservedRunningTime="2026-11-02 20:47:20.510659605 +0000 UTC m=+3521.508348208"
Nov 02 20:47:23.833484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:23.833446    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:26.253068 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:26.253036    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"
Nov 02 20:47:27.261889 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:27.261862    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"
Nov 02 20:47:27.358341 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:47:27.358304    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:47:28.712813 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:28.712754    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:29.109789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:29.107988    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"]
Nov 02 20:47:29.165395 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:29.165359    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:31.289358 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:31.289328    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"
Nov 02 20:47:31.289838 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:31.289401    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"
Nov 02 20:47:33.611306 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:33.611268    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:38.375700 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:38.375653    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:43.207152 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:43.207106    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:43.288096 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:43.288062    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:47:44.128570 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.128513    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95" podUID="02bb97cd-3492-4849-ae6f-3b4481e7ca42" containerName="main" containerID="cri-o://259826522cad6706753e153dd43d1506021f0ce28b23a399e564ca3a31250d34" gracePeriod=45
Nov 02 20:47:44.394956 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.394880    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:47:44.420545 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.420505    5224 generic.go:356] "Generic (PLEG): container finished" podID="02bb97cd-3492-4849-ae6f-3b4481e7ca42" containerID="259826522cad6706753e153dd43d1506021f0ce28b23a399e564ca3a31250d34" exitCode=0
Nov 02 20:47:44.420749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.420556    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"
Nov 02 20:47:44.420749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.420603    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95" event={"ID":"02bb97cd-3492-4849-ae6f-3b4481e7ca42","Type":"ContainerDied","Data":"259826522cad6706753e153dd43d1506021f0ce28b23a399e564ca3a31250d34"}
Nov 02 20:47:44.420749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.420663    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95" event={"ID":"02bb97cd-3492-4849-ae6f-3b4481e7ca42","Type":"ContainerDied","Data":"5843fe69bb7321579a111935ea81807ed72654a8abd2fdc2d5cd76d085b2be15"}
Nov 02 20:47:44.420749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.420690    5224 scope.go:122] "RemoveContainer" containerID="259826522cad6706753e153dd43d1506021f0ce28b23a399e564ca3a31250d34"
Nov 02 20:47:44.432941 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.432870    5224 scope.go:122] "RemoveContainer" containerID="259826522cad6706753e153dd43d1506021f0ce28b23a399e564ca3a31250d34"
Nov 02 20:47:44.433290 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:47:44.433261    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259826522cad6706753e153dd43d1506021f0ce28b23a399e564ca3a31250d34\": container with ID starting with 259826522cad6706753e153dd43d1506021f0ce28b23a399e564ca3a31250d34 not found: ID does not exist" containerID="259826522cad6706753e153dd43d1506021f0ce28b23a399e564ca3a31250d34"
Nov 02 20:47:44.433375 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.433303    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259826522cad6706753e153dd43d1506021f0ce28b23a399e564ca3a31250d34"} err="failed to get container status \"259826522cad6706753e153dd43d1506021f0ce28b23a399e564ca3a31250d34\": rpc error: code = NotFound desc = could not find container \"259826522cad6706753e153dd43d1506021f0ce28b23a399e564ca3a31250d34\": container with ID starting with 259826522cad6706753e153dd43d1506021f0ce28b23a399e564ca3a31250d34 not found: ID does not exist"
Nov 02 20:47:44.472781 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.472736    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bb97cd-3492-4849-ae6f-3b4481e7ca42-kube-api-access-jnlbj\" (UniqueName: \"kubernetes.io/projected/02bb97cd-3492-4849-ae6f-3b4481e7ca42-kube-api-access-jnlbj\") pod \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") "
Nov 02 20:47:44.472944 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.472795    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tmp-dir\") pod \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") "
Nov 02 20:47:44.472944 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.472930    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-dshm\") pod \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") "
Nov 02 20:47:44.473033 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.472959    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tls-certs\" (UniqueName: \"kubernetes.io/secret/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tls-certs\") pod \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") "
Nov 02 20:47:44.473033 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.472979    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-model-cache\") pod \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") "
Nov 02 20:47:44.473125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.473020    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tmp-dir" pod "02bb97cd-3492-4849-ae6f-3b4481e7ca42" (UID: "02bb97cd-3492-4849-ae6f-3b4481e7ca42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:47:44.473125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.473045    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-home\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-home\") pod \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\" (UID: \"02bb97cd-3492-4849-ae6f-3b4481e7ca42\") "
Nov 02 20:47:44.473244 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.473195    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-model-cache" pod "02bb97cd-3492-4849-ae6f-3b4481e7ca42" (UID: "02bb97cd-3492-4849-ae6f-3b4481e7ca42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:47:44.473349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.473317    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-home" pod "02bb97cd-3492-4849-ae6f-3b4481e7ca42" (UID: "02bb97cd-3492-4849-ae6f-3b4481e7ca42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:47:44.473483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.473444    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:44.473483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.473462    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:44.473483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.473472    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:44.475073 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.475048    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-dshm" pod "02bb97cd-3492-4849-ae6f-3b4481e7ca42" (UID: "02bb97cd-3492-4849-ae6f-3b4481e7ca42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:47:44.475197 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.475107    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tls-certs" pod "02bb97cd-3492-4849-ae6f-3b4481e7ca42" (UID: "02bb97cd-3492-4849-ae6f-3b4481e7ca42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:47:44.475257 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.475205    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bb97cd-3492-4849-ae6f-3b4481e7ca42-kube-api-access-jnlbj" pod "02bb97cd-3492-4849-ae6f-3b4481e7ca42" (UID: "02bb97cd-3492-4849-ae6f-3b4481e7ca42"). InnerVolumeSpecName "kube-api-access-jnlbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:47:44.574546 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.574513    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnlbj\" (UniqueName: \"kubernetes.io/projected/02bb97cd-3492-4849-ae6f-3b4481e7ca42-kube-api-access-jnlbj\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:44.574546 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.574541    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bb97cd-3492-4849-ae6f-3b4481e7ca42-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:44.574546 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.574550    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02bb97cd-3492-4849-ae6f-3b4481e7ca42-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:44.844821 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.844792    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"]
Nov 02 20:47:44.846750 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:44.846730    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-09ecd0/auth-test-model-09ecd0-kserve-f9c4dd6fc-rrv95"]
Nov 02 20:47:45.167649 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:45.167616    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02bb97cd-3492-4849-ae6f-3b4481e7ca42" path="/var/lib/kubelet/pods/02bb97cd-3492-4849-ae6f-3b4481e7ca42/volumes"
Nov 02 20:47:48.250242 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:48.250198    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:53.267489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:53.267436    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:47:57.358509 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:47:57.358474    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:47:58.249125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:47:58.249081    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:03.274184 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:03.274145    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:04.728371 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:04.728339    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:05.411989 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:05.411953    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:08.185148 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:08.185114    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:12.344982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:12.344940    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:13.063672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:13.063636    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:14.797197 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:14.797150    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:16.013988 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.013949    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:16.459502 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.459425    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"]
Nov 02 20:48:16.459960 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.459946    5224 cpu_manager.go:414] "containerMap: removing container" podUID="02bb97cd-3492-4849-ae6f-3b4481e7ca42" containerName="main"
Nov 02 20:48:16.460017 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.459961    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bb97cd-3492-4849-ae6f-3b4481e7ca42" containerName="main"
Nov 02 20:48:16.460064 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.460055    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bb97cd-3492-4849-ae6f-3b4481e7ca42" containerName="main"
Nov 02 20:48:16.463305 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.463288    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.517410 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.517368    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-credential-socket\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.517676 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.517654    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-podinfo\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.517829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.517812    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2424b57-879c-47f4-b608-6e7731fcefaf-istiod-ca-cert\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.517986 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.517968    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-socket\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.518132 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.518116    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-certs\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.519016 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.518328    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-token\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.519016 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.518576    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg4qf\" (UniqueName: \"kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-kube-api-access-dg4qf\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.519016 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.518654    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-data\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.519016 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.518783    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-envoy\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.525852 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.525829    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-020389-openshift-default-dockercfg-nj4wb\""
Nov 02 20:48:16.574922 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.574883    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"]
Nov 02 20:48:16.586813 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.586781    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:16.619863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.619816    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4qf\" (UniqueName: \"kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-kube-api-access-dg4qf\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.620075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.619889    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-data\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.620075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.619923    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-envoy\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.620075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.619959    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-credential-socket\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.620075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.619979    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-podinfo\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.620075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.620005    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2424b57-879c-47f4-b608-6e7731fcefaf-istiod-ca-cert\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.620075 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.620039    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-socket\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.620422 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.620082    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-certs\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.620422 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.620113    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-token\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.620422 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.620391    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-data\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.620558 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.620415    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-credential-socket\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.620558 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.620516    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-certs\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.620754 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.620706    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-socket\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.621421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.621395    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2424b57-879c-47f4-b608-6e7731fcefaf-istiod-ca-cert\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.623234 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.623206    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-podinfo\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.623454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.623433    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-envoy\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.678970 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.678930    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-token\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.679169 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.679054    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4qf\" (UniqueName: \"kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-kube-api-access-dg4qf\") pod \"e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") " pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.776758 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.776717    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:16.922038 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:16.921995    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:17.202170 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:48:17.202137    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2424b57_879c_47f4_b608_6e7731fcefaf.slice/crio-a76bce61b4bf01fc55dd2530c78b2cd98b163c86a507008ebd47818b511666f5 WatchSource:0}: Error finding container a76bce61b4bf01fc55dd2530c78b2cd98b163c86a507008ebd47818b511666f5: Status 404 returned error can't find the container with id a76bce61b4bf01fc55dd2530c78b2cd98b163c86a507008ebd47818b511666f5
Nov 02 20:48:17.205314 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:17.205272    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:48:17.205414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:17.205359    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:48:17.205414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:17.205405    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:48:17.205553 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:17.205535    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"]
Nov 02 20:48:17.632689 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:17.632581    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" event={"ID":"d2424b57-879c-47f4-b608-6e7731fcefaf","Type":"ContainerStarted","Data":"d629c6e5981416881b80483bacf7daffaf2b3e863f568f8a6019e4629a3e8657"}
Nov 02 20:48:17.632689 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:17.632652    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" event={"ID":"d2424b57-879c-47f4-b608-6e7731fcefaf","Type":"ContainerStarted","Data":"a76bce61b4bf01fc55dd2530c78b2cd98b163c86a507008ebd47818b511666f5"}
Nov 02 20:48:17.750825 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:17.750754    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" podStartSLOduration=1.750740669 podStartE2EDuration="1.750740669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:17.74954278 +0000 UTC m=+3578.747231389" watchObservedRunningTime="2026-11-02 20:48:17.750740669 +0000 UTC m=+3578.748429276"
Nov 02 20:48:17.777478 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:17.777451    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:18.472534 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:18.472493    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:18.782858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:18.782829    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:19.643560 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:19.643525    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:19.644531 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:19.644512    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:48:20.590968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:20.590938    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:20.812922 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:20.812888    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:20.991010 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:20.990922    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:22.113269 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:22.113233    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:22.722823 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:22.722784    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:23.161589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:23.161550    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:23.853237 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:23.853193    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:24.048859 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.048821    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:24.207039 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.206961    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"]
Nov 02 20:48:24.211215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.211196    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"
Nov 02 20:48:24.265375 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.265342    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"]
Nov 02 20:48:24.306909 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.306865    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz\" (UID: \"89904cd2-94b1-4b5a-b7ee-f070a316ec4e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"
Nov 02 20:48:24.307266 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.307242    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wqz\" (UniqueName: \"kubernetes.io/projected/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-kube-api-access-z5wqz\") pod \"payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz\" (UID: \"89904cd2-94b1-4b5a-b7ee-f070a316ec4e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"
Nov 02 20:48:24.313000 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.312965    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-020389-dockercfg-wvvgj\""
Nov 02 20:48:24.313111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.313073    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-020389\""
Nov 02 20:48:24.375676 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.375645    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"]
Nov 02 20:48:24.380419 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.380391    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"
Nov 02 20:48:24.408516 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.408489    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz\" (UID: \"89904cd2-94b1-4b5a-b7ee-f070a316ec4e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"
Nov 02 20:48:24.408667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.408527    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wqz\" (UniqueName: \"kubernetes.io/projected/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-kube-api-access-z5wqz\") pod \"payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz\" (UID: \"89904cd2-94b1-4b5a-b7ee-f070a316ec4e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"
Nov 02 20:48:24.409167 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.409145    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz\" (UID: \"89904cd2-94b1-4b5a-b7ee-f070a316ec4e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"
Nov 02 20:48:24.432455 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.432419    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"]
Nov 02 20:48:24.467053 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.466976    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wqz\" (UniqueName: \"kubernetes.io/projected/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-kube-api-access-z5wqz\") pod \"payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz\" (UID: \"89904cd2-94b1-4b5a-b7ee-f070a316ec4e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"
Nov 02 20:48:24.509983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.509950    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4e5727-61d0-472f-b24f-d264e21e4783-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4\" (UID: \"9e4e5727-61d0-472f-b24f-d264e21e4783\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"
Nov 02 20:48:24.510141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.510108    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktnsm\" (UniqueName: \"kubernetes.io/projected/9e4e5727-61d0-472f-b24f-d264e21e4783-kube-api-access-ktnsm\") pod \"payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4\" (UID: \"9e4e5727-61d0-472f-b24f-d264e21e4783\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"
Nov 02 20:48:24.525702 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.525669    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"
Nov 02 20:48:24.611749 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.611707    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4e5727-61d0-472f-b24f-d264e21e4783-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4\" (UID: \"9e4e5727-61d0-472f-b24f-d264e21e4783\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"
Nov 02 20:48:24.611929 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.611913    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktnsm\" (UniqueName: \"kubernetes.io/projected/9e4e5727-61d0-472f-b24f-d264e21e4783-kube-api-access-ktnsm\") pod \"payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4\" (UID: \"9e4e5727-61d0-472f-b24f-d264e21e4783\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"
Nov 02 20:48:24.612547 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.612502    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4e5727-61d0-472f-b24f-d264e21e4783-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4\" (UID: \"9e4e5727-61d0-472f-b24f-d264e21e4783\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"
Nov 02 20:48:24.670175 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.670144    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktnsm\" (UniqueName: \"kubernetes.io/projected/9e4e5727-61d0-472f-b24f-d264e21e4783-kube-api-access-ktnsm\") pod \"payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4\" (UID: \"9e4e5727-61d0-472f-b24f-d264e21e4783\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"
Nov 02 20:48:24.694283 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.694260    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"
Nov 02 20:48:24.890915 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:48:24.890882    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89904cd2_94b1_4b5a_b7ee_f070a316ec4e.slice/crio-bc17ed950c173fd07aea83a367f7ce0077e718e941c00c580c07d1cd71a512aa WatchSource:0}: Error finding container bc17ed950c173fd07aea83a367f7ce0077e718e941c00c580c07d1cd71a512aa: Status 404 returned error can't find the container with id bc17ed950c173fd07aea83a367f7ce0077e718e941c00c580c07d1cd71a512aa
Nov 02 20:48:24.894502 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:24.894479    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"]
Nov 02 20:48:25.060892 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:48:25.060863    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4e5727_61d0_472f_b24f_d264e21e4783.slice/crio-83d00a48fbc1c482e20fedab0f016f0b0cd0e57b6219201907070cfcc832cf0b WatchSource:0}: Error finding container 83d00a48fbc1c482e20fedab0f016f0b0cd0e57b6219201907070cfcc832cf0b: Status 404 returned error can't find the container with id 83d00a48fbc1c482e20fedab0f016f0b0cd0e57b6219201907070cfcc832cf0b
Nov 02 20:48:25.064375 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:25.064356    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"]
Nov 02 20:48:25.679392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:25.679338    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz" event={"ID":"89904cd2-94b1-4b5a-b7ee-f070a316ec4e","Type":"ContainerStarted","Data":"06a9734293de106e4ccf09e3e30ed5154d67655155f43771b3b2af49ba6c2f52"}
Nov 02 20:48:25.679860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:25.679412    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"
Nov 02 20:48:25.679860 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:25.679438    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz" event={"ID":"89904cd2-94b1-4b5a-b7ee-f070a316ec4e","Type":"ContainerStarted","Data":"bc17ed950c173fd07aea83a367f7ce0077e718e941c00c580c07d1cd71a512aa"}
Nov 02 20:48:25.681012 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:25.680986    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4" event={"ID":"9e4e5727-61d0-472f-b24f-d264e21e4783","Type":"ContainerStarted","Data":"7b967f1e31d45d5357c4be6bb889bc62d1a9f32b1d298a5f467a056b934b08e5"}
Nov 02 20:48:25.681157 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:25.681036    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4" event={"ID":"9e4e5727-61d0-472f-b24f-d264e21e4783","Type":"ContainerStarted","Data":"83d00a48fbc1c482e20fedab0f016f0b0cd0e57b6219201907070cfcc832cf0b"}
Nov 02 20:48:25.681157 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:25.681066    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"
Nov 02 20:48:25.796247 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:25.796191    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz" podStartSLOduration=1.7961739410000002 podStartE2EDuration="1.796173941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:25.794351953 +0000 UTC m=+3586.792040560" watchObservedRunningTime="2026-11-02 20:48:25.796173941 +0000 UTC m=+3586.793862552"
Nov 02 20:48:25.912019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:25.911943    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4" podStartSLOduration=1.911917618 podStartE2EDuration="1.911917618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:25.909279589 +0000 UTC m=+3586.906968200" watchObservedRunningTime="2026-11-02 20:48:25.911917618 +0000 UTC m=+3586.909606228"
Nov 02 20:48:26.880600 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:26.880558    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:27.107242 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:27.107207    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:27.358879 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:48:27.358847    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:48:29.143166 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:29.143127    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:29.314189 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:29.314144    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:29.482123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:29.482040    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:32.137598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:32.137555    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:32.350902 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:32.350861    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:34.058574 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.058540    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"]
Nov 02 20:48:34.062729 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.062712    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.115447 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.115411    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"]
Nov 02 20:48:34.163787 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.163740    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-720bda-serving-cert\""
Nov 02 20:48:34.224877 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.224846    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2805b48b-58b8-486e-b3bb-63b0aedbbb05-openshift-service-ca\") pod \"maas-api-e2e-autores-720bda-55d6f58f58-cdz7l\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.225065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.224883    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-maas-api-tls\") pod \"maas-api-e2e-autores-720bda-55d6f58f58-cdz7l\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.225137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.225108    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-metrics-certs\") pod \"maas-api-e2e-autores-720bda-55d6f58f58-cdz7l\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.225194 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.225163    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnbw\" (UniqueName: \"kubernetes.io/projected/2805b48b-58b8-486e-b3bb-63b0aedbbb05-kube-api-access-sbnbw\") pod \"maas-api-e2e-autores-720bda-55d6f58f58-cdz7l\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.326095 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.326014    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-maas-api-tls\") pod \"maas-api-e2e-autores-720bda-55d6f58f58-cdz7l\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.326270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.326136    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-metrics-certs\") pod \"maas-api-e2e-autores-720bda-55d6f58f58-cdz7l\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.326270 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:48:34.326164    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-720bda-serving-cert: secret "maas-api-e2e-autores-720bda-serving-cert" not found
Nov 02 20:48:34.326270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.326166    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnbw\" (UniqueName: \"kubernetes.io/projected/2805b48b-58b8-486e-b3bb-63b0aedbbb05-kube-api-access-sbnbw\") pod \"maas-api-e2e-autores-720bda-55d6f58f58-cdz7l\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.326270 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:48:34.326227    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-maas-api-tls podName:2805b48b-58b8-486e-b3bb-63b0aedbbb05 nodeName:}" failed. No retries permitted until 2026-11-02 20:48:34.826210839 +0000 UTC m=+3595.823899425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-maas-api-tls") pod "maas-api-e2e-autores-720bda-55d6f58f58-cdz7l" (UID: "2805b48b-58b8-486e-b3bb-63b0aedbbb05") : secret "maas-api-e2e-autores-720bda-serving-cert" not found
Nov 02 20:48:34.326481 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.326359    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2805b48b-58b8-486e-b3bb-63b0aedbbb05-openshift-service-ca\") pod \"maas-api-e2e-autores-720bda-55d6f58f58-cdz7l\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.326945 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.326927    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2805b48b-58b8-486e-b3bb-63b0aedbbb05-openshift-service-ca\") pod \"maas-api-e2e-autores-720bda-55d6f58f58-cdz7l\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.328537 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.328516    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-metrics-certs\") pod \"maas-api-e2e-autores-720bda-55d6f58f58-cdz7l\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.350480 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.350439    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:34.385447 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.385415    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnbw\" (UniqueName: \"kubernetes.io/projected/2805b48b-58b8-486e-b3bb-63b0aedbbb05-kube-api-access-sbnbw\") pod \"maas-api-e2e-autores-720bda-55d6f58f58-cdz7l\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.564994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.564959    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:34.731542 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.731438    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:34.831492 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.831457    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-maas-api-tls\") pod \"maas-api-e2e-autores-720bda-55d6f58f58-cdz7l\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.833995 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.833967    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-maas-api-tls\") pod \"maas-api-e2e-autores-720bda-55d6f58f58-cdz7l\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:34.976113 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:34.976077    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:35.349046 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:48:35.349007    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2805b48b_58b8_486e_b3bb_63b0aedbbb05.slice/crio-58e6cca01ae1816e3055619e51192b2153a0677fd2fafb7e330722705eb5200a WatchSource:0}: Error finding container 58e6cca01ae1816e3055619e51192b2153a0677fd2fafb7e330722705eb5200a: Status 404 returned error can't find the container with id 58e6cca01ae1816e3055619e51192b2153a0677fd2fafb7e330722705eb5200a
Nov 02 20:48:35.350843 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:35.350819    5224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:48:35.353255 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:35.353231    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"]
Nov 02 20:48:35.738649 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:35.738571    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l" event={"ID":"2805b48b-58b8-486e-b3bb-63b0aedbbb05","Type":"ContainerStarted","Data":"58e6cca01ae1816e3055619e51192b2153a0677fd2fafb7e330722705eb5200a"}
Nov 02 20:48:36.688888 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:36.688858    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"
Nov 02 20:48:36.689260 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:36.689014    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"
Nov 02 20:48:36.746221 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:36.746188    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l" event={"ID":"2805b48b-58b8-486e-b3bb-63b0aedbbb05","Type":"ContainerStarted","Data":"9b0be14be89bdb2d26d687183e3cb1362810c91aebc576983f9a1b339e8d6151"}
Nov 02 20:48:36.746390 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:36.746309    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:36.919982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:36.919925    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l" podStartSLOduration=2.126003737 podStartE2EDuration="2.919904692s" totalImagesPullingTime="793.900955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:36.917981534 +0000 UTC m=+3597.915670142" watchObservedRunningTime="2026-11-02 20:48:36.919904692 +0000 UTC m=+3597.917593301"
Nov 02 20:48:39.051731 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.051701    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"]
Nov 02 20:48:39.055695 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.055678    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.080549 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.080520    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.080663 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.080557    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4586a4d6-e64d-480e-bf09-64fb701d239e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.080705 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.080666    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-metrics-certs\") pod \"maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.080741 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.080711    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqcsn\" (UniqueName: \"kubernetes.io/projected/4586a4d6-e64d-480e-bf09-64fb701d239e-kube-api-access-tqcsn\") pod \"maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.109296 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.109271    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"]
Nov 02 20:48:39.157857 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.157832    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-608c11-serving-cert\""
Nov 02 20:48:39.181314 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.181288    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.181481 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.181336    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4586a4d6-e64d-480e-bf09-64fb701d239e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.181481 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.181394    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-metrics-certs\") pod \"maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.181481 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:48:39.181444    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-serving-cert: secret "maas-api-e2e-extmod-a-608c11-serving-cert" not found
Nov 02 20:48:39.181647 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.181448    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqcsn\" (UniqueName: \"kubernetes.io/projected/4586a4d6-e64d-480e-bf09-64fb701d239e-kube-api-access-tqcsn\") pod \"maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.181647 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:48:39.181511    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-maas-api-tls podName:4586a4d6-e64d-480e-bf09-64fb701d239e nodeName:}" failed. No retries permitted until 2026-11-02 20:48:39.681492202 +0000 UTC m=+3600.679180801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-maas-api-tls") pod "maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc" (UID: "4586a4d6-e64d-480e-bf09-64fb701d239e") : secret "maas-api-e2e-extmod-a-608c11-serving-cert" not found
Nov 02 20:48:39.182033 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.182016    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4586a4d6-e64d-480e-bf09-64fb701d239e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.183647 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.183625    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-metrics-certs\") pod \"maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.245489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.245462    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqcsn\" (UniqueName: \"kubernetes.io/projected/4586a4d6-e64d-480e-bf09-64fb701d239e-kube-api-access-tqcsn\") pod \"maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.290374 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.290339    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:39.500139 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.500051    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:39.675715 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.675675    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:39.688018 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.687985    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.690244 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.690226    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:39.968317 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:39.968284    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:40.341937 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:48:40.341904    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4586a4d6_e64d_480e_bf09_64fb701d239e.slice/crio-4c30f1300eb88f085823e56717debac7515a34412425f4fddd764b03262bbdc1 WatchSource:0}: Error finding container 4c30f1300eb88f085823e56717debac7515a34412425f4fddd764b03262bbdc1: Status 404 returned error can't find the container with id 4c30f1300eb88f085823e56717debac7515a34412425f4fddd764b03262bbdc1
Nov 02 20:48:40.345708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:40.345685    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"]
Nov 02 20:48:40.770231 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:40.770192    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc" event={"ID":"4586a4d6-e64d-480e-bf09-64fb701d239e","Type":"ContainerStarted","Data":"4c30f1300eb88f085823e56717debac7515a34412425f4fddd764b03262bbdc1"}
Nov 02 20:48:41.778056 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:41.778019    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc" event={"ID":"4586a4d6-e64d-480e-bf09-64fb701d239e","Type":"ContainerStarted","Data":"34b980d610bbea90b0e8c60596b69d6142741357a59a486b827d654c02e90e13"}
Nov 02 20:48:41.778493 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:41.778164    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:41.902194 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:41.902146    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc" podStartSLOduration=2.066737391 podStartE2EDuration="2.90213331s" totalImagesPullingTime="835.395919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:41.899939239 +0000 UTC m=+3602.897627847" watchObservedRunningTime="2026-11-02 20:48:41.90213331 +0000 UTC m=+3602.899821918"
Nov 02 20:48:42.757274 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:42.757241    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:48:44.363643 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:44.363573    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:47.789392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:47.789360    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:48:49.228421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:49.228368    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:54.055986 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:54.055952    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:56.353210 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:56.353166    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:48:57.357529 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:48:57.357494    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:48:58.962216 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:48:58.962185    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:03.904854 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:03.904814    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:04.117182 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:04.117140    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:49:05.057590 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:05.057541    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:05.790264 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:05.790228    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:08.789096 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:08.789060    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:13.637994 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:13.637955    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:18.543171 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:18.543131    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:23.457568 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:23.457534    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:27.359606 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:49:27.359567    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:49:28.614784 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:28.614740    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:33.316921 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:33.316881    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:38.149795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:38.149739    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:43.014180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:43.014141    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:47.784090 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:47.784050    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:52.595241 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:52.595202    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:57.358470 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:49:57.358430    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:49:57.416965 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:57.416928    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:57.611789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:57.611688    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:49:57.778506 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:49:57.778466    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:02.562115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:02.562073    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:05.383585 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:05.383549    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:06.307541 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:06.307500    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:07.119528 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:07.119488    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:07.340445 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:07.340408    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:07.513098 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:07.513071    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:07.690634 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:07.690595    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:10.828085 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:10.828048    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:12.369607 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:12.369567    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:12.860213 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:12.860171    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:13.080482 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:13.080449    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:13.969189 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:13.969153    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"]
Nov 02 20:50:14.076413 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:14.076371    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"]
Nov 02 20:50:14.349497 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:14.349459    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:14.524551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:14.524512    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:16.488712 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.488666    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"]
Nov 02 20:50:16.489281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.489032    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l" podUID="2805b48b-58b8-486e-b3bb-63b0aedbbb05" containerName="maas-api" containerID="cri-o://9b0be14be89bdb2d26d687183e3cb1362810c91aebc576983f9a1b339e8d6151" gracePeriod=30
Nov 02 20:50:16.733171 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.733131    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:16.761141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.761113    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:50:16.886200 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.885298    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-metrics-certs\") pod \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") "
Nov 02 20:50:16.886200 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.885431    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2805b48b-58b8-486e-b3bb-63b0aedbbb05-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2805b48b-58b8-486e-b3bb-63b0aedbbb05-openshift-service-ca\") pod \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") "
Nov 02 20:50:16.886200 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.885562    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2805b48b-58b8-486e-b3bb-63b0aedbbb05-kube-api-access-sbnbw\" (UniqueName: \"kubernetes.io/projected/2805b48b-58b8-486e-b3bb-63b0aedbbb05-kube-api-access-sbnbw\") pod \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") "
Nov 02 20:50:16.886200 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.885639    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-maas-api-tls\") pod \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\" (UID: \"2805b48b-58b8-486e-b3bb-63b0aedbbb05\") "
Nov 02 20:50:16.893794 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.889205    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-maas-api-tls" pod "2805b48b-58b8-486e-b3bb-63b0aedbbb05" (UID: "2805b48b-58b8-486e-b3bb-63b0aedbbb05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:50:16.903140 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.903086    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-metrics-certs" pod "2805b48b-58b8-486e-b3bb-63b0aedbbb05" (UID: "2805b48b-58b8-486e-b3bb-63b0aedbbb05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:50:16.903387 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.903354    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2805b48b-58b8-486e-b3bb-63b0aedbbb05-openshift-service-ca" pod "2805b48b-58b8-486e-b3bb-63b0aedbbb05" (UID: "2805b48b-58b8-486e-b3bb-63b0aedbbb05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:50:16.905098 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.905073    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2805b48b-58b8-486e-b3bb-63b0aedbbb05-kube-api-access-sbnbw" pod "2805b48b-58b8-486e-b3bb-63b0aedbbb05" (UID: "2805b48b-58b8-486e-b3bb-63b0aedbbb05"). InnerVolumeSpecName "kube-api-access-sbnbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:50:16.986885 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.986842    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbnbw\" (UniqueName: \"kubernetes.io/projected/2805b48b-58b8-486e-b3bb-63b0aedbbb05-kube-api-access-sbnbw\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:16.986885 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.986877    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:16.986885 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.986892    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2805b48b-58b8-486e-b3bb-63b0aedbbb05-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:16.987204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:16.986905    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2805b48b-58b8-486e-b3bb-63b0aedbbb05-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:17.062374 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:17.062331    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:17.362206 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:17.362104    5224 generic.go:356] "Generic (PLEG): container finished" podID="2805b48b-58b8-486e-b3bb-63b0aedbbb05" containerID="9b0be14be89bdb2d26d687183e3cb1362810c91aebc576983f9a1b339e8d6151" exitCode=0
Nov 02 20:50:17.362206 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:17.362149    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"
Nov 02 20:50:17.362206 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:17.362166    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l" event={"ID":"2805b48b-58b8-486e-b3bb-63b0aedbbb05","Type":"ContainerDied","Data":"9b0be14be89bdb2d26d687183e3cb1362810c91aebc576983f9a1b339e8d6151"}
Nov 02 20:50:17.362519 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:17.362223    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l" event={"ID":"2805b48b-58b8-486e-b3bb-63b0aedbbb05","Type":"ContainerDied","Data":"58e6cca01ae1816e3055619e51192b2153a0677fd2fafb7e330722705eb5200a"}
Nov 02 20:50:17.362519 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:17.362255    5224 scope.go:122] "RemoveContainer" containerID="9b0be14be89bdb2d26d687183e3cb1362810c91aebc576983f9a1b339e8d6151"
Nov 02 20:50:17.373407 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:17.373383    5224 scope.go:122] "RemoveContainer" containerID="9b0be14be89bdb2d26d687183e3cb1362810c91aebc576983f9a1b339e8d6151"
Nov 02 20:50:17.373759 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:50:17.373737    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b0be14be89bdb2d26d687183e3cb1362810c91aebc576983f9a1b339e8d6151\": container with ID starting with 9b0be14be89bdb2d26d687183e3cb1362810c91aebc576983f9a1b339e8d6151 not found: ID does not exist" containerID="9b0be14be89bdb2d26d687183e3cb1362810c91aebc576983f9a1b339e8d6151"
Nov 02 20:50:17.373834 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:17.373788    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b0be14be89bdb2d26d687183e3cb1362810c91aebc576983f9a1b339e8d6151"} err="failed to get container status \"9b0be14be89bdb2d26d687183e3cb1362810c91aebc576983f9a1b339e8d6151\": rpc error: code = NotFound desc = could not find container \"9b0be14be89bdb2d26d687183e3cb1362810c91aebc576983f9a1b339e8d6151\": container with ID starting with 9b0be14be89bdb2d26d687183e3cb1362810c91aebc576983f9a1b339e8d6151 not found: ID does not exist"
Nov 02 20:50:17.530238 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:17.530203    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"]
Nov 02 20:50:17.580692 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:17.580651    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-720bda-55d6f58f58-cdz7l"]
Nov 02 20:50:18.335387 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:18.335299    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:50:18.404735 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:18.404705    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:18.674551 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:18.674461    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:18.905731 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:18.905694    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:18.974663 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:18.972851    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t" podUID="4b6258ad-7932-485c-aece-f41b6fda4c6e" containerName="payload-processing" containerID="cri-o://9876ce01baf720beb90286d53334eab30db3db517aafea3457eb1970cd56afdb" gracePeriod=25
Nov 02 20:50:19.077164 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.077114    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s" podUID="01316187-77b6-4788-b77f-72b8482f42a0" containerName="payload-pre-processing" containerID="cri-o://f20aafdea0cad06afd68ecaf3d45b44e41e2c6c20b513ade5b6694c65e7a6c33" gracePeriod=25
Nov 02 20:50:19.185306 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.185270    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2805b48b-58b8-486e-b3bb-63b0aedbbb05" path="/var/lib/kubelet/pods/2805b48b-58b8-486e-b3bb-63b0aedbbb05/volumes"
Nov 02 20:50:19.385256 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.385224    5224 generic.go:356] "Generic (PLEG): container finished" podID="4b6258ad-7932-485c-aece-f41b6fda4c6e" containerID="9876ce01baf720beb90286d53334eab30db3db517aafea3457eb1970cd56afdb" exitCode=0
Nov 02 20:50:19.385434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.385304    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"
Nov 02 20:50:19.385434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.385325    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t" event={"ID":"4b6258ad-7932-485c-aece-f41b6fda4c6e","Type":"ContainerDied","Data":"9876ce01baf720beb90286d53334eab30db3db517aafea3457eb1970cd56afdb"}
Nov 02 20:50:19.385434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.385373    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t" event={"ID":"4b6258ad-7932-485c-aece-f41b6fda4c6e","Type":"ContainerDied","Data":"501c9da4c28cdf5446b28f8f28f9911f14b76cb12c4efdb85c1c39837ae8b02a"}
Nov 02 20:50:19.385434 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.385391    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="501c9da4c28cdf5446b28f8f28f9911f14b76cb12c4efdb85c1c39837ae8b02a"
Nov 02 20:50:19.387000 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.386978    5224 generic.go:356] "Generic (PLEG): container finished" podID="01316187-77b6-4788-b77f-72b8482f42a0" containerID="f20aafdea0cad06afd68ecaf3d45b44e41e2c6c20b513ade5b6694c65e7a6c33" exitCode=0
Nov 02 20:50:19.387127 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.387066    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s" event={"ID":"01316187-77b6-4788-b77f-72b8482f42a0","Type":"ContainerDied","Data":"f20aafdea0cad06afd68ecaf3d45b44e41e2c6c20b513ade5b6694c65e7a6c33"}
Nov 02 20:50:19.428269 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.428244    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"
Nov 02 20:50:19.516011 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.515979    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01316187-77b6-4788-b77f-72b8482f42a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01316187-77b6-4788-b77f-72b8482f42a0-plugins-config-volume\") pod \"01316187-77b6-4788-b77f-72b8482f42a0\" (UID: \"01316187-77b6-4788-b77f-72b8482f42a0\") "
Nov 02 20:50:19.516207 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.516090    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6258ad-7932-485c-aece-f41b6fda4c6e-kube-api-access-4h7dq\" (UniqueName: \"kubernetes.io/projected/4b6258ad-7932-485c-aece-f41b6fda4c6e-kube-api-access-4h7dq\") pod \"4b6258ad-7932-485c-aece-f41b6fda4c6e\" (UID: \"4b6258ad-7932-485c-aece-f41b6fda4c6e\") "
Nov 02 20:50:19.516283 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.516266    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01316187-77b6-4788-b77f-72b8482f42a0-kube-api-access-qwchq\" (UniqueName: \"kubernetes.io/projected/01316187-77b6-4788-b77f-72b8482f42a0-kube-api-access-qwchq\") pod \"01316187-77b6-4788-b77f-72b8482f42a0\" (UID: \"01316187-77b6-4788-b77f-72b8482f42a0\") "
Nov 02 20:50:19.516376 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.516355    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b6258ad-7932-485c-aece-f41b6fda4c6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6258ad-7932-485c-aece-f41b6fda4c6e-plugins-config-volume\") pod \"4b6258ad-7932-485c-aece-f41b6fda4c6e\" (UID: \"4b6258ad-7932-485c-aece-f41b6fda4c6e\") "
Nov 02 20:50:19.516448 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.516427    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01316187-77b6-4788-b77f-72b8482f42a0-plugins-config-volume" pod "01316187-77b6-4788-b77f-72b8482f42a0" (UID: "01316187-77b6-4788-b77f-72b8482f42a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:50:19.516791 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.516756    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01316187-77b6-4788-b77f-72b8482f42a0-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:19.516918 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.516818    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6258ad-7932-485c-aece-f41b6fda4c6e-plugins-config-volume" pod "4b6258ad-7932-485c-aece-f41b6fda4c6e" (UID: "4b6258ad-7932-485c-aece-f41b6fda4c6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:50:19.518258 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.518237    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6258ad-7932-485c-aece-f41b6fda4c6e-kube-api-access-4h7dq" pod "4b6258ad-7932-485c-aece-f41b6fda4c6e" (UID: "4b6258ad-7932-485c-aece-f41b6fda4c6e"). InnerVolumeSpecName "kube-api-access-4h7dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:50:19.518319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.518287    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01316187-77b6-4788-b77f-72b8482f42a0-kube-api-access-qwchq" pod "01316187-77b6-4788-b77f-72b8482f42a0" (UID: "01316187-77b6-4788-b77f-72b8482f42a0"). InnerVolumeSpecName "kube-api-access-qwchq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:50:19.617987 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.617947    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwchq\" (UniqueName: \"kubernetes.io/projected/01316187-77b6-4788-b77f-72b8482f42a0-kube-api-access-qwchq\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:19.617987 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.617979    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6258ad-7932-485c-aece-f41b6fda4c6e-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:19.617987 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:19.617989    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h7dq\" (UniqueName: \"kubernetes.io/projected/4b6258ad-7932-485c-aece-f41b6fda4c6e-kube-api-access-4h7dq\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:20.127116 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:20.127076    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:20.302862 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:20.302816    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:20.394897 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:20.394866    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"
Nov 02 20:50:20.395092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:20.394902    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s" event={"ID":"01316187-77b6-4788-b77f-72b8482f42a0","Type":"ContainerDied","Data":"8c81e7adeae4bea0267fdcde2dfe338fc9720f0a5d619e3c2b686df881e1560f"}
Nov 02 20:50:20.395092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:20.394974    5224 scope.go:122] "RemoveContainer" containerID="f20aafdea0cad06afd68ecaf3d45b44e41e2c6c20b513ade5b6694c65e7a6c33"
Nov 02 20:50:20.395232 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:20.395197    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"
Nov 02 20:50:20.544822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:20.544762    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:20.571328 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:20.571225    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"]
Nov 02 20:50:20.621632 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:20.621594    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-720bda-5d685f599f-lkf4s"]
Nov 02 20:50:20.735736 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:20.735691    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"]
Nov 02 20:50:20.744197 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:20.744160    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-720bda-765b646cfc-9954t"]
Nov 02 20:50:21.135875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:21.135831    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:21.177242 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:21.177195    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01316187-77b6-4788-b77f-72b8482f42a0" path="/var/lib/kubelet/pods/01316187-77b6-4788-b77f-72b8482f42a0/volumes"
Nov 02 20:50:21.177705 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:21.177682    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b6258ad-7932-485c-aece-f41b6fda4c6e" path="/var/lib/kubelet/pods/4b6258ad-7932-485c-aece-f41b6fda4c6e/volumes"
Nov 02 20:50:21.898677 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:21.898633    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"]
Nov 02 20:50:22.012021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:22.011985    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"]
Nov 02 20:50:22.283497 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:22.283462    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:22.503655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:22.503619    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:22.675524 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:22.675438    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:23.115755 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.115717    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:23.613088 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.613049    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"]
Nov 02 20:50:23.613580 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.613551    5224 cpu_manager.go:414] "containerMap: removing container" podUID="4b6258ad-7932-485c-aece-f41b6fda4c6e" containerName="payload-processing"
Nov 02 20:50:23.613580 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.613563    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6258ad-7932-485c-aece-f41b6fda4c6e" containerName="payload-processing"
Nov 02 20:50:23.613580 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.613576    5224 cpu_manager.go:414] "containerMap: removing container" podUID="2805b48b-58b8-486e-b3bb-63b0aedbbb05" containerName="maas-api"
Nov 02 20:50:23.613580 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.613581    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2805b48b-58b8-486e-b3bb-63b0aedbbb05" containerName="maas-api"
Nov 02 20:50:23.613720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.613599    5224 cpu_manager.go:414] "containerMap: removing container" podUID="01316187-77b6-4788-b77f-72b8482f42a0" containerName="payload-pre-processing"
Nov 02 20:50:23.613720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.613604    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01316187-77b6-4788-b77f-72b8482f42a0" containerName="payload-pre-processing"
Nov 02 20:50:23.613720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.613695    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="01316187-77b6-4788-b77f-72b8482f42a0" containerName="payload-pre-processing"
Nov 02 20:50:23.613720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.613710    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6258ad-7932-485c-aece-f41b6fda4c6e" containerName="payload-processing"
Nov 02 20:50:23.613720 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.613717    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="2805b48b-58b8-486e-b3bb-63b0aedbbb05" containerName="maas-api"
Nov 02 20:50:23.618354 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.618329    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.703948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.703915    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:23.706646 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.706604    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"]
Nov 02 20:50:23.759178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.759132    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-28197c-openshift-default-dockercfg-bqhs8\""
Nov 02 20:50:23.793430 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.793392    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-socket\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.793714 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.793693    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istiod-ca-cert\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.793889 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.793873    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-certs\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.797943 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.797907    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-envoy\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.798222 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.798205    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-token\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.798426 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.798410    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-data\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.798584 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.798568    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-podinfo\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.798737 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.798724    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-credential-socket\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.798894 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.798880    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blftx\" (UniqueName: \"kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-kube-api-access-blftx\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.878433 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.878330    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:23.900670 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.900623    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-socket\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.900899 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.900690    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istiod-ca-cert\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.900899 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.900760    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-certs\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.900899 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.900835    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-envoy\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.900899 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.900860    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-token\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.901146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.900954    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-data\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.901146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.901003    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-podinfo\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.901146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.901060    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-credential-socket\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.901146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.901099    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blftx\" (UniqueName: \"kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-kube-api-access-blftx\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.901146 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.901104    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-socket\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.901421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.901151    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-certs\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.901528 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.901507    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-credential-socket\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.901631 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.901529    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istiod-ca-cert\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.901631 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.901529    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-data\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.903421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.903398    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-envoy\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.903548 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.903531    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-podinfo\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.961691 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.961617    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-token\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:23.964499 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:23.964465    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blftx\" (UniqueName: \"kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-kube-api-access-blftx\") pod \"e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") " pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:24.066457 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:24.066370    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:24.232542 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:24.232449    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:24.657317 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:50:24.657282    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67dd13d0_2c6c_4838_ba70_dc6053151dc7.slice/crio-51c2f944a179a8ea7f723fd6e66efb603e2c26ea65763bc5032d4132b9d96ec4 WatchSource:0}: Error finding container 51c2f944a179a8ea7f723fd6e66efb603e2c26ea65763bc5032d4132b9d96ec4: Status 404 returned error can't find the container with id 51c2f944a179a8ea7f723fd6e66efb603e2c26ea65763bc5032d4132b9d96ec4
Nov 02 20:50:24.660405 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:24.660363    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:50:24.660503 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:24.660455    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:50:24.660503 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:24.660493    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:50:24.660659 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:24.660639    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"]
Nov 02 20:50:25.431390 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:25.431343    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7" event={"ID":"67dd13d0-2c6c-4838-ba70-dc6053151dc7","Type":"ContainerStarted","Data":"fe99b22f92154447f569cd4bde2919973652fba5e47028f64f0cf5c8a21ced36"}
Nov 02 20:50:25.431583 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:25.431416    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7" event={"ID":"67dd13d0-2c6c-4838-ba70-dc6053151dc7","Type":"ContainerStarted","Data":"51c2f944a179a8ea7f723fd6e66efb603e2c26ea65763bc5032d4132b9d96ec4"}
Nov 02 20:50:25.651250 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:25.651186    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7" podStartSLOduration=2.651164637 podStartE2EDuration="2.651164637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:50:25.599811859 +0000 UTC m=+3706.597500468" watchObservedRunningTime="2026-11-02 20:50:25.651164637 +0000 UTC m=+3706.648853246"
Nov 02 20:50:26.233113 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:26.233075    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:26.239403 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:26.239375    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:26.437594 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:26.437562    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:26.438689 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:26.438667    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:50:26.899356 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:26.899308    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj" podUID="87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267" containerName="payload-processing" containerID="cri-o://6da135b40f664c0b1f62892c72ac770c4753af694aae691825daf7f5d2fd71b3" gracePeriod=25
Nov 02 20:50:27.012879 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.012828    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh" podUID="cd6be353-2b67-4544-89fd-96c6233ac47e" containerName="payload-pre-processing" containerID="cri-o://3b9b4965a37d5be5c52834ec213e0a8d7ec57de4416d5b802060db26b0b60669" gracePeriod=25
Nov 02 20:50:27.179962 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.179927    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"
Nov 02 20:50:27.253514 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.253482    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-kube-api-access-fgxjj\" (UniqueName: \"kubernetes.io/projected/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-kube-api-access-fgxjj\") pod \"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267\" (UID: \"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267\") "
Nov 02 20:50:27.253982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.253540    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-plugins-config-volume\") pod \"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267\" (UID: \"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267\") "
Nov 02 20:50:27.254249 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.254221    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-plugins-config-volume" pod "87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267" (UID: "87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:50:27.255921 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.255889    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-kube-api-access-fgxjj" pod "87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267" (UID: "87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267"). InnerVolumeSpecName "kube-api-access-fgxjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:50:27.276889 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.276855    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"
Nov 02 20:50:27.354817 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.354779    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6be353-2b67-4544-89fd-96c6233ac47e-kube-api-access-kb55d\" (UniqueName: \"kubernetes.io/projected/cd6be353-2b67-4544-89fd-96c6233ac47e-kube-api-access-kb55d\") pod \"cd6be353-2b67-4544-89fd-96c6233ac47e\" (UID: \"cd6be353-2b67-4544-89fd-96c6233ac47e\") "
Nov 02 20:50:27.355019 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.354959    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6be353-2b67-4544-89fd-96c6233ac47e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6be353-2b67-4544-89fd-96c6233ac47e-plugins-config-volume\") pod \"cd6be353-2b67-4544-89fd-96c6233ac47e\" (UID: \"cd6be353-2b67-4544-89fd-96c6233ac47e\") "
Nov 02 20:50:27.355305 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.355273    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6be353-2b67-4544-89fd-96c6233ac47e-plugins-config-volume" pod "cd6be353-2b67-4544-89fd-96c6233ac47e" (UID: "cd6be353-2b67-4544-89fd-96c6233ac47e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:50:27.355487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.355395    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6be353-2b67-4544-89fd-96c6233ac47e-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:27.355487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.355408    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgxjj\" (UniqueName: \"kubernetes.io/projected/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-kube-api-access-fgxjj\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:27.355487 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.355419    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:27.357039 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.357015    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6be353-2b67-4544-89fd-96c6233ac47e-kube-api-access-kb55d" pod "cd6be353-2b67-4544-89fd-96c6233ac47e" (UID: "cd6be353-2b67-4544-89fd-96c6233ac47e"). InnerVolumeSpecName "kube-api-access-kb55d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:50:27.362247 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:50:27.362215    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:50:27.445901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.445803    5224 generic.go:356] "Generic (PLEG): container finished" podID="cd6be353-2b67-4544-89fd-96c6233ac47e" containerID="3b9b4965a37d5be5c52834ec213e0a8d7ec57de4416d5b802060db26b0b60669" exitCode=0
Nov 02 20:50:27.445901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.445840    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh" event={"ID":"cd6be353-2b67-4544-89fd-96c6233ac47e","Type":"ContainerDied","Data":"3b9b4965a37d5be5c52834ec213e0a8d7ec57de4416d5b802060db26b0b60669"}
Nov 02 20:50:27.445901 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.445870    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"
Nov 02 20:50:27.446205 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.445907    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh" event={"ID":"cd6be353-2b67-4544-89fd-96c6233ac47e","Type":"ContainerDied","Data":"016f61cc00acb7c2fb1a4ad4ea500457776aa2d406f7aab9849f16eda3e3df02"}
Nov 02 20:50:27.446205 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.445938    5224 scope.go:122] "RemoveContainer" containerID="3b9b4965a37d5be5c52834ec213e0a8d7ec57de4416d5b802060db26b0b60669"
Nov 02 20:50:27.448873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.448845    5224 generic.go:356] "Generic (PLEG): container finished" podID="87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267" containerID="6da135b40f664c0b1f62892c72ac770c4753af694aae691825daf7f5d2fd71b3" exitCode=0
Nov 02 20:50:27.452922 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.449039    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"
Nov 02 20:50:27.452922 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.449104    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj" event={"ID":"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267","Type":"ContainerDied","Data":"6da135b40f664c0b1f62892c72ac770c4753af694aae691825daf7f5d2fd71b3"}
Nov 02 20:50:27.452922 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.449162    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj" event={"ID":"87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267","Type":"ContainerDied","Data":"552998f838a8cc827580a1803c8ce23e72d5ef788f0a81b8180dfb9c500a63c9"}
Nov 02 20:50:27.456407 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.456368    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb55d\" (UniqueName: \"kubernetes.io/projected/cd6be353-2b67-4544-89fd-96c6233ac47e-kube-api-access-kb55d\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:27.460419 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.460395    5224 scope.go:122] "RemoveContainer" containerID="3b9b4965a37d5be5c52834ec213e0a8d7ec57de4416d5b802060db26b0b60669"
Nov 02 20:50:27.460806 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:50:27.460782    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9b4965a37d5be5c52834ec213e0a8d7ec57de4416d5b802060db26b0b60669\": container with ID starting with 3b9b4965a37d5be5c52834ec213e0a8d7ec57de4416d5b802060db26b0b60669 not found: ID does not exist" containerID="3b9b4965a37d5be5c52834ec213e0a8d7ec57de4416d5b802060db26b0b60669"
Nov 02 20:50:27.460916 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.460813    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9b4965a37d5be5c52834ec213e0a8d7ec57de4416d5b802060db26b0b60669"} err="failed to get container status \"3b9b4965a37d5be5c52834ec213e0a8d7ec57de4416d5b802060db26b0b60669\": rpc error: code = NotFound desc = could not find container \"3b9b4965a37d5be5c52834ec213e0a8d7ec57de4416d5b802060db26b0b60669\": container with ID starting with 3b9b4965a37d5be5c52834ec213e0a8d7ec57de4416d5b802060db26b0b60669 not found: ID does not exist"
Nov 02 20:50:27.460916 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.460835    5224 scope.go:122] "RemoveContainer" containerID="6da135b40f664c0b1f62892c72ac770c4753af694aae691825daf7f5d2fd71b3"
Nov 02 20:50:27.472410 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.472380    5224 scope.go:122] "RemoveContainer" containerID="6da135b40f664c0b1f62892c72ac770c4753af694aae691825daf7f5d2fd71b3"
Nov 02 20:50:27.472729 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:50:27.472707    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da135b40f664c0b1f62892c72ac770c4753af694aae691825daf7f5d2fd71b3\": container with ID starting with 6da135b40f664c0b1f62892c72ac770c4753af694aae691825daf7f5d2fd71b3 not found: ID does not exist" containerID="6da135b40f664c0b1f62892c72ac770c4753af694aae691825daf7f5d2fd71b3"
Nov 02 20:50:27.472830 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.472743    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da135b40f664c0b1f62892c72ac770c4753af694aae691825daf7f5d2fd71b3"} err="failed to get container status \"6da135b40f664c0b1f62892c72ac770c4753af694aae691825daf7f5d2fd71b3\": rpc error: code = NotFound desc = could not find container \"6da135b40f664c0b1f62892c72ac770c4753af694aae691825daf7f5d2fd71b3\": container with ID starting with 6da135b40f664c0b1f62892c72ac770c4753af694aae691825daf7f5d2fd71b3 not found: ID does not exist"
Nov 02 20:50:27.670988 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.670954    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"]
Nov 02 20:50:27.673694 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.673671    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-608c11-66955bc577-fpbkj"]
Nov 02 20:50:27.784539 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.784503    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"]
Nov 02 20:50:27.787021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:27.786995    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-608c11-779fcc7947-fvxdh"]
Nov 02 20:50:28.377909 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:28.377874    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:28.457827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:28.457791    5224 generic.go:356] "Generic (PLEG): container finished" podID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerID="d629c6e5981416881b80483bacf7daffaf2b3e863f568f8a6019e4629a3e8657" exitCode=0
Nov 02 20:50:28.458001 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:28.457864    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" event={"ID":"d2424b57-879c-47f4-b608-6e7731fcefaf","Type":"ContainerDied","Data":"d629c6e5981416881b80483bacf7daffaf2b3e863f568f8a6019e4629a3e8657"}
Nov 02 20:50:28.458318 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:28.458297    5224 scope.go:122] "RemoveContainer" containerID="d629c6e5981416881b80483bacf7daffaf2b3e863f568f8a6019e4629a3e8657"
Nov 02 20:50:28.459944 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:28.459907    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:50:28.460057 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:28.459985    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:50:28.460057 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:28.460021    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:50:28.559692 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:28.559641    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:28.736429 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:28.736313    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:29.174454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:29.174390    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:29.188251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:29.188216    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267" path="/var/lib/kubelet/pods/87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267/volumes"
Nov 02 20:50:29.188679 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:29.188660    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd6be353-2b67-4544-89fd-96c6233ac47e" path="/var/lib/kubelet/pods/cd6be353-2b67-4544-89fd-96c6233ac47e/volumes"
Nov 02 20:50:29.479595 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:29.479501    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" event={"ID":"d2424b57-879c-47f4-b608-6e7731fcefaf","Type":"ContainerStarted","Data":"9f808f76869abecee8e792be1912899e1c1f8c532b5281048c3dc63b4a291290"}
Nov 02 20:50:29.777677 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:29.777638    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:50:29.777977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:29.777950    5224 patch_prober.go:29] interesting pod/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 20:50:29.778043 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:29.778012    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 20:50:30.777490 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:30.777451    5224 patch_prober.go:29] interesting pod/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 20:50:30.777951 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:30.777514    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 20:50:31.526881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:31.526848    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:31.700484 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:31.700446    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:31.779855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:31.779163    5224 patch_prober.go:29] interesting pod/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 20:50:31.779855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:31.779240    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 20:50:32.777413 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:32.777371    5224 patch_prober.go:29] interesting pod/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 20:50:32.777615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:32.777432    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 20:50:33.475473 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:33.475432    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:33.777371 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:33.777335    5224 patch_prober.go:29] interesting pod/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 20:50:33.777569 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:33.777400    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 20:50:34.268883 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:34.258263    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:34.643692 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:34.643598    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:50:34.777165 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:34.777129    5224 patch_prober.go:29] interesting pod/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 20:50:34.777346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:34.777195    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 20:50:35.782756 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:35.782724    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:50:35.783751 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:35.783730    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:50:39.066593 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:39.066552    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:43.122746 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:43.122705    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:43.881346 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:43.881305    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:45.598152 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:45.598117    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:48.758783 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:48.758723    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:51.990646 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:51.990604    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:50:53.677997 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:53.677951    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:50:57.358846 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:50:57.358807    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:50:58.577196 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:50:58.577155    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:03.464463 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:03.464417    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:05.715844 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:05.715809    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:06.459124 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:06.459092    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:08.339405 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:08.339364    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:13.162500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:13.162462    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:18.009713 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:18.009671    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:20.936610 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:20.936561    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:21.151622 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:21.151586    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:22.892234 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:22.892199    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:25.217184 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:25.217146    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:27.358542 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:51:27.358503    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:51:27.769042 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:27.768999    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:30.783421 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:30.783386    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:31.514894 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:31.514863    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:31.917362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:31.917266    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:32.087894 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:32.087854    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:32.944455 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:32.944415    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:36.524945 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:36.524900    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:37.396195 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:37.396151    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:37.999103 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:37.999060    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:38.735591 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:38.735539    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:38.911579 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:38.911542    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:42.997385 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:42.997345    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:43.343340 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:43.343300    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:48.761235 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:48.761150    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:53.225940 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:53.225900    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:54.411699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.411663    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"]
Nov 02 20:51:54.412151 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.411921    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc" podUID="4586a4d6-e64d-480e-bf09-64fb701d239e" containerName="maas-api" containerID="cri-o://34b980d610bbea90b0e8c60596b69d6142741357a59a486b827d654c02e90e13" gracePeriod=30
Nov 02 20:51:54.607806 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.607727    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:54.692817 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.692698    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:51:54.861562 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.861521    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-maas-api-tls\") pod \"4586a4d6-e64d-480e-bf09-64fb701d239e\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") "
Nov 02 20:51:54.861562 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.861572    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4586a4d6-e64d-480e-bf09-64fb701d239e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4586a4d6-e64d-480e-bf09-64fb701d239e-openshift-service-ca\") pod \"4586a4d6-e64d-480e-bf09-64fb701d239e\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") "
Nov 02 20:51:54.861853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.861706    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-metrics-certs\") pod \"4586a4d6-e64d-480e-bf09-64fb701d239e\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") "
Nov 02 20:51:54.861853 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.861743    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4586a4d6-e64d-480e-bf09-64fb701d239e-kube-api-access-tqcsn\" (UniqueName: \"kubernetes.io/projected/4586a4d6-e64d-480e-bf09-64fb701d239e-kube-api-access-tqcsn\") pod \"4586a4d6-e64d-480e-bf09-64fb701d239e\" (UID: \"4586a4d6-e64d-480e-bf09-64fb701d239e\") "
Nov 02 20:51:54.862006 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.861965    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4586a4d6-e64d-480e-bf09-64fb701d239e-openshift-service-ca" pod "4586a4d6-e64d-480e-bf09-64fb701d239e" (UID: "4586a4d6-e64d-480e-bf09-64fb701d239e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:51:54.862211 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.862178    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4586a4d6-e64d-480e-bf09-64fb701d239e-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:51:54.863982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.863953    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4586a4d6-e64d-480e-bf09-64fb701d239e-kube-api-access-tqcsn" pod "4586a4d6-e64d-480e-bf09-64fb701d239e" (UID: "4586a4d6-e64d-480e-bf09-64fb701d239e"). InnerVolumeSpecName "kube-api-access-tqcsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:51:54.863982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.863958    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-metrics-certs" pod "4586a4d6-e64d-480e-bf09-64fb701d239e" (UID: "4586a4d6-e64d-480e-bf09-64fb701d239e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:51:54.864158 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.864100    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-maas-api-tls" pod "4586a4d6-e64d-480e-bf09-64fb701d239e" (UID: "4586a4d6-e64d-480e-bf09-64fb701d239e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:51:54.963148 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.963070    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:51:54.963148 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.963100    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqcsn\" (UniqueName: \"kubernetes.io/projected/4586a4d6-e64d-480e-bf09-64fb701d239e-kube-api-access-tqcsn\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:51:54.963148 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:54.963111    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4586a4d6-e64d-480e-bf09-64fb701d239e-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:51:55.026269 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:55.026207    5224 generic.go:356] "Generic (PLEG): container finished" podID="4586a4d6-e64d-480e-bf09-64fb701d239e" containerID="34b980d610bbea90b0e8c60596b69d6142741357a59a486b827d654c02e90e13" exitCode=0
Nov 02 20:51:55.026486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:55.026393    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc" event={"ID":"4586a4d6-e64d-480e-bf09-64fb701d239e","Type":"ContainerDied","Data":"34b980d610bbea90b0e8c60596b69d6142741357a59a486b827d654c02e90e13"}
Nov 02 20:51:55.026486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:55.026444    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc" event={"ID":"4586a4d6-e64d-480e-bf09-64fb701d239e","Type":"ContainerDied","Data":"4c30f1300eb88f085823e56717debac7515a34412425f4fddd764b03262bbdc1"}
Nov 02 20:51:55.026486 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:55.026477    5224 scope.go:122] "RemoveContainer" containerID="34b980d610bbea90b0e8c60596b69d6142741357a59a486b827d654c02e90e13"
Nov 02 20:51:55.026655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:55.026640    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"
Nov 02 20:51:55.043162 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:55.043133    5224 scope.go:122] "RemoveContainer" containerID="34b980d610bbea90b0e8c60596b69d6142741357a59a486b827d654c02e90e13"
Nov 02 20:51:55.043460 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:51:55.043435    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b980d610bbea90b0e8c60596b69d6142741357a59a486b827d654c02e90e13\": container with ID starting with 34b980d610bbea90b0e8c60596b69d6142741357a59a486b827d654c02e90e13 not found: ID does not exist" containerID="34b980d610bbea90b0e8c60596b69d6142741357a59a486b827d654c02e90e13"
Nov 02 20:51:55.043556 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:55.043468    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b980d610bbea90b0e8c60596b69d6142741357a59a486b827d654c02e90e13"} err="failed to get container status \"34b980d610bbea90b0e8c60596b69d6142741357a59a486b827d654c02e90e13\": rpc error: code = NotFound desc = could not find container \"34b980d610bbea90b0e8c60596b69d6142741357a59a486b827d654c02e90e13\": container with ID starting with 34b980d610bbea90b0e8c60596b69d6142741357a59a486b827d654c02e90e13 not found: ID does not exist"
Nov 02 20:51:55.124935 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:55.124896    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:55.199850 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:55.199817    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"]
Nov 02 20:51:55.202652 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:55.202623    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-608c11-7564cbd6d5-gqrwc"]
Nov 02 20:51:55.591251 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:55.591217    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:56.477374 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:56.477335    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:56.692699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:56.692666    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:56.888304 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:56.888271    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:57.167989 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:57.167896    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4586a4d6-e64d-480e-bf09-64fb701d239e" path="/var/lib/kubelet/pods/4586a4d6-e64d-480e-bf09-64fb701d239e/volumes"
Nov 02 20:51:57.358675 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:51:57.358641    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:51:58.937117 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:58.937071    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:51:59.147305 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:51:59.147266    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:01.232132 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.232089    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:01.456788 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.455388    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:01.755808 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.755760    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"]
Nov 02 20:52:01.756311 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.756296    5224 cpu_manager.go:414] "containerMap: removing container" podUID="cd6be353-2b67-4544-89fd-96c6233ac47e" containerName="payload-pre-processing"
Nov 02 20:52:01.756357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.756313    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6be353-2b67-4544-89fd-96c6233ac47e" containerName="payload-pre-processing"
Nov 02 20:52:01.756357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.756334    5224 cpu_manager.go:414] "containerMap: removing container" podUID="87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267" containerName="payload-processing"
Nov 02 20:52:01.756357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.756340    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267" containerName="payload-processing"
Nov 02 20:52:01.756357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.756354    5224 cpu_manager.go:414] "containerMap: removing container" podUID="4586a4d6-e64d-480e-bf09-64fb701d239e" containerName="maas-api"
Nov 02 20:52:01.756489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.756360    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4586a4d6-e64d-480e-bf09-64fb701d239e" containerName="maas-api"
Nov 02 20:52:01.756489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.756465    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6be353-2b67-4544-89fd-96c6233ac47e" containerName="payload-pre-processing"
Nov 02 20:52:01.756489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.756476    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d4b8d6-1ca9-4eb0-ba0d-3a5ba813c267" containerName="payload-processing"
Nov 02 20:52:01.756489 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.756487    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="4586a4d6-e64d-480e-bf09-64fb701d239e" containerName="maas-api"
Nov 02 20:52:01.760194 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.760178    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:01.811722 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.811688    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"]
Nov 02 20:52:01.848270 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.848241    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8wj\" (UniqueName: \"kubernetes.io/projected/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-kube-api-access-cx8wj\") pod \"maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:01.848456 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.848317    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:01.848514 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.848456    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-metrics-certs\") pod \"maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:01.848514 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.848503    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:01.860454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.860428    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-28197c-serving-cert\""
Nov 02 20:52:01.949562 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.949528    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-metrics-certs\") pod \"maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:01.949757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.949584    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:01.949757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.949662    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8wj\" (UniqueName: \"kubernetes.io/projected/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-kube-api-access-cx8wj\") pod \"maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:01.949757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.949720    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:01.949981 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:52:01.949825    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-serving-cert: secret "maas-api-e2e-extmod-a-28197c-serving-cert" not found
Nov 02 20:52:01.949981 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:52:01.949880    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-maas-api-tls podName:4351f7c6-66c0-447b-bd89-e2d35d3d43b7 nodeName:}" failed. No retries permitted until 2026-11-02 20:52:02.449866717 +0000 UTC m=+3803.447555304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-maas-api-tls") pod "maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v" (UID: "4351f7c6-66c0-447b-bd89-e2d35d3d43b7") : secret "maas-api-e2e-extmod-a-28197c-serving-cert" not found
Nov 02 20:52:01.950315 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.950295    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:01.952212 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:01.952190    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-metrics-certs\") pod \"maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:02.004213 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:02.004174    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:02.011006 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:02.010941    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8wj\" (UniqueName: \"kubernetes.io/projected/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-kube-api-access-cx8wj\") pod \"maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:02.177225 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:02.177184    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:02.350589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:02.350552    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:02.458454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:02.458426    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:02.460822 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:02.460792    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:02.672047 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:02.671962    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:02.690352 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:02.690308    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:02.870137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:02.870099    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:03.046947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:03.046600    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"]
Nov 02 20:52:03.077208 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:03.077176    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v" event={"ID":"4351f7c6-66c0-447b-bd89-e2d35d3d43b7","Type":"ContainerStarted","Data":"fcddc16a7276c7fae86a76c7fa7fe316187b6e696010f22a5155ed0291d73341"}
Nov 02 20:52:04.086111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:04.086066    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v" event={"ID":"4351f7c6-66c0-447b-bd89-e2d35d3d43b7","Type":"ContainerStarted","Data":"a2808f16d1fb32be8ed4bb797a3a6a7b89a289e39fc2c4aec473b78a8a912b4e"}
Nov 02 20:52:04.086542 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:04.086233    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:04.240522 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:04.240415    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v" podStartSLOduration=2.349476932 podStartE2EDuration="3.240394565s" totalImagesPullingTime="890.917633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:52:04.235454853 +0000 UTC m=+3805.233143464" watchObservedRunningTime="2026-11-02 20:52:04.240394565 +0000 UTC m=+3805.238083175"
Nov 02 20:52:04.270664 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:04.270625    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:04.437968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:04.437930    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:06.070241 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:06.070202    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:06.803682 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:06.803648    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:07.149105 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:07.149019    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:10.097850 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:10.097818    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:52:12.004362 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:12.004317    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:12.238612 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:12.238579    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:52:17.028203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:17.028163    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:22.226129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:22.226083    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:27.156158 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.156118    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:27.298298 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.298258    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"]
Nov 02 20:52:27.301072 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.301049    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.360720 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:52:27.360673    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:52:27.383684 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.383654    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"]
Nov 02 20:52:27.435995 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.435906    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.435995 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.435990    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.436214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.436059    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.436214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.436101    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.436214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.436206    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.436340 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.436280    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.436340 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.436335    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmxld\" (UniqueName: \"kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-kube-api-access-pmxld\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.436444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.436431    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.436483 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.436454    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.447454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.447435    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-phngj\""
Nov 02 20:52:27.537182 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.537150    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.537374 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.537199    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.537374 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.537279    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.537374 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.537326    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.537555 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.537390    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmxld\" (UniqueName: \"kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-kube-api-access-pmxld\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.537555 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.537471    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.537555 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.537499    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.537555 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.537539    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.537797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.537582    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.537797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.537663    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.537797 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.537688    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.537985 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.537881    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.537985 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.537955    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.538312 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.538290    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.539604 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.539586    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.539844 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.539826    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.596889 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.596859    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.596999 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.596983    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmxld\" (UniqueName: \"kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-kube-api-access-pmxld\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-59gzc\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:27.613508 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:27.613484    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:28.025704 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:52:28.025675    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695b48a2_b9f5_4cb9_8e40_26835cdb5d71.slice/crio-c863d657db4335e27a5c2166a7d49359cd3145dccc5cd1a338a0b24d99761d9b WatchSource:0}: Error finding container c863d657db4335e27a5c2166a7d49359cd3145dccc5cd1a338a0b24d99761d9b: Status 404 returned error can't find the container with id c863d657db4335e27a5c2166a7d49359cd3145dccc5cd1a338a0b24d99761d9b
Nov 02 20:52:28.028243 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:28.028216    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:52:28.028319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:28.028279    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:52:28.028319 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:28.028311    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:52:28.029407 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:28.029388    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"]
Nov 02 20:52:28.240719 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:28.240686    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" event={"ID":"695b48a2-b9f5-4cb9-8e40-26835cdb5d71","Type":"ContainerStarted","Data":"5ff6edc901f6778d89f1248ac472d5e86e58ae6ca1c1376427c33bbbf473b47c"}
Nov 02 20:52:28.241231 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:28.240743    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" event={"ID":"695b48a2-b9f5-4cb9-8e40-26835cdb5d71","Type":"ContainerStarted","Data":"c863d657db4335e27a5c2166a7d49359cd3145dccc5cd1a338a0b24d99761d9b"}
Nov 02 20:52:28.359153 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:28.358882    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" podStartSLOduration=1.358864787 podStartE2EDuration="1.358864787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:52:28.357559567 +0000 UTC m=+3829.355248176" watchObservedRunningTime="2026-11-02 20:52:28.358864787 +0000 UTC m=+3829.356553397"
Nov 02 20:52:28.614332 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:28.614252    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:29.615490 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:29.615454    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-59gzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 02 20:52:29.615977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:29.615512    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" podUID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 02 20:52:30.614161 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:30.614126    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-59gzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 02 20:52:30.614333 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:30.614178    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" podUID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 02 20:52:31.614850 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:31.614807    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-59gzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 02 20:52:31.615352 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:31.614874    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" podUID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 02 20:52:32.044519 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:32.044464    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:32.614082 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:32.614047    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-59gzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 02 20:52:32.614262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:32.614106    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" podUID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 02 20:52:33.614694 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:33.614664    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-59gzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 02 20:52:33.615123 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:33.614713    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" podUID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 02 20:52:34.614359 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:34.614314    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-59gzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 02 20:52:34.614582 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:34.614383    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" podUID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 02 20:52:35.614829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:35.614761    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-59gzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 02 20:52:35.615254 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:35.614856    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" podUID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 02 20:52:36.845052 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:36.845016    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:36.845501 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:36.845316    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:36.846156 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:36.846129    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 20:52:36.886971 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:36.886942    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:41.807175 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:41.807121    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:46.651616 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:46.651576    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:51.495637 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:51.495512    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:56.411071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:52:56.410998    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:52:57.380400 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:52:57.380360    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:53:00.829226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:00.829121    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:02.681824 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:02.681536    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:03.081537 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:03.081427    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:03.593402 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:03.593361    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:04.275505 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:04.275468    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:04.512039 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:04.511940    5224 generic.go:356] "Generic (PLEG): container finished" podID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerID="fe99b22f92154447f569cd4bde2919973652fba5e47028f64f0cf5c8a21ced36" exitCode=0
Nov 02 20:53:04.512039 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:04.512010    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7" event={"ID":"67dd13d0-2c6c-4838-ba70-dc6053151dc7","Type":"ContainerDied","Data":"fe99b22f92154447f569cd4bde2919973652fba5e47028f64f0cf5c8a21ced36"}
Nov 02 20:53:04.512449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:04.512434    5224 scope.go:122] "RemoveContainer" containerID="fe99b22f92154447f569cd4bde2919973652fba5e47028f64f0cf5c8a21ced36"
Nov 02 20:53:04.513989 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:04.513957    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:53:04.514116 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:04.514040    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:53:04.514116 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:04.514090    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:53:05.519470 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:05.519435    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7" event={"ID":"67dd13d0-2c6c-4838-ba70-dc6053151dc7","Type":"ContainerStarted","Data":"aaa59d408159bf0844eefb6200f13cc63761a7cca30b7985f1cc8337525d8ca6"}
Nov 02 20:53:06.233035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:06.233005    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:53:06.233335 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:06.233314    5224 patch_prober.go:29] interesting pod/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 02 20:53:06.233431 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:06.233359    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7" podUID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 02 20:53:06.299559 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:06.299521    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:06.485569 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:06.485456    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:07.233599 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:07.233567    5224 patch_prober.go:29] interesting pod/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 02 20:53:07.234035 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:07.233628    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7" podUID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 02 20:53:07.361810 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:07.361585    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:08.075467 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:08.075423    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:08.233321 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:08.233287    5224 patch_prober.go:29] interesting pod/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 02 20:53:08.233532 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:08.233354    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7" podUID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 02 20:53:09.233541 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:09.233507    5224 patch_prober.go:29] interesting pod/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 02 20:53:09.233982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:09.233573    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7" podUID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 02 20:53:10.218808 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:10.218747    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:10.233845 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:10.233813    5224 patch_prober.go:29] interesting pod/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 02 20:53:10.234218 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:10.233874    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7" podUID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 02 20:53:11.238174 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:11.238143    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:53:11.238567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:11.238407    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:53:11.239071 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:11.239053    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:53:11.269052 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:11.269014    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:13.513445 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:13.513404    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:15.712969 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:15.712928    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:16.860188 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:16.860132    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:17.749786 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:17.749727    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:17.860072 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:17.860034    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:53:19.565289 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:19.565200    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:21.433658 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:21.433617    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:21.617200 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:21.617164    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:23.019511 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:23.019465    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:23.487046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:23.486941    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:24.668811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:24.668752    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:25.086132 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:25.086092    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:25.260359 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:25.260324    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:27.358374 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:53:27.358344    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:53:28.116418 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:28.116374    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:30.040152 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:30.040110    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:33.933031 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:33.932980    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:35.055971 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:35.055932    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:35.541370 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:35.541330    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:35.754349 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:35.754316    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:39.376968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:39.376935    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:39.952272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:39.952227    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:40.669780 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:40.669734    5224 scope.go:122] "RemoveContainer" containerID="9876ce01baf720beb90286d53334eab30db3db517aafea3457eb1970cd56afdb"
Nov 02 20:53:44.859415 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:44.859376    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:49.756255 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:49.756216    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:54.643631 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:54.643583    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:53:57.359002 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:53:57.358961    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:53:59.494996 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:53:59.494945    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:04.524215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:04.524171    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:06.719324 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:06.719288    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:07.478157 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:07.478121    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:09.366022 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:09.365976    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:14.258369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:14.258331    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:16.404980 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:16.404932    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:16.650577 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:16.650534    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:17.613637 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:17.613598    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:54:19.145502 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:19.145459    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:19.342672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:19.342640    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:19.512333 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:19.512241    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:19.933479 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:19.933435    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:20.105754 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:20.105720    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:21.690517 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:21.690478    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:22.372709 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:22.370681    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:22.777053 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:22.777017    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:23.490344 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:23.490303    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:25.299868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:25.299833    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:25.489041 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:25.489002    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:25.682671 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:25.682587    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:25.878353 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:25.878312    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:26.404837 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:26.404804    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:26.581202 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:26.581169    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:27.099564 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.099511    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:27.346755 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.346714    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:27.358541 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:54:27.358467    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:54:27.626742 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.626641    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:27.896666 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.896589    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"]
Nov 02 20:54:27.901533 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.901513    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:27.963469 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.963425    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:27.963662 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.963480    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:27.963662 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.963540    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3542c449-0d85-4534-9284-67c7250ccd64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:27.963662 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.963602    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:27.963886 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.963692    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:27.963886 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.963713    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:27.963886 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.963748    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:27.963886 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.963824    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3542c449-0d85-4534-9284-67c7250ccd64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:27.963886 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.963853    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n4th\" (UniqueName: \"kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-kube-api-access-9n4th\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:27.987197 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:27.987170    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"]
Nov 02 20:54:28.065201 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.065167    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.065201 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.065206    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.065416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.065247    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.065416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.065275    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3542c449-0d85-4534-9284-67c7250ccd64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.065416 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.065313    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n4th\" (UniqueName: \"kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-kube-api-access-9n4th\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.065611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.065464    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.065611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.065517    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.065611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.065542    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3542c449-0d85-4534-9284-67c7250ccd64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.065611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.065570    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.065611 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.065593    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.065892 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.065634    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.065892 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.065822    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.066004 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.065966    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.066089 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.066068    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3542c449-0d85-4534-9284-67c7250ccd64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.067701 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.067670    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.067915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.067899    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3542c449-0d85-4534-9284-67c7250ccd64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.124399 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.124367    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.124594 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.124560    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n4th\" (UniqueName: \"kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-kube-api-access-9n4th\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-gc9b5\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.216569 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.216469    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:28.535635 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.535594    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:28.635500 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:54:28.635455    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3542c449_0d85_4534_9284_67c7250ccd64.slice/crio-562ea3a7dc37023cd28efa3316f24a26413cbedf4908e018b3576bd04e9905d3 WatchSource:0}: Error finding container 562ea3a7dc37023cd28efa3316f24a26413cbedf4908e018b3576bd04e9905d3: Status 404 returned error can't find the container with id 562ea3a7dc37023cd28efa3316f24a26413cbedf4908e018b3576bd04e9905d3
Nov 02 20:54:28.638005 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.637967    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:54:28.638096 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.638036    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:54:28.638096 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.638064    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:54:28.638333 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:28.638314    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"]
Nov 02 20:54:29.038948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:29.038253    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" event={"ID":"3542c449-0d85-4534-9284-67c7250ccd64","Type":"ContainerStarted","Data":"78712418b6bf4ffd1d160982b72a164881906739011c2622413a536f41cace63"}
Nov 02 20:54:29.038948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:29.038316    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" event={"ID":"3542c449-0d85-4534-9284-67c7250ccd64","Type":"ContainerStarted","Data":"562ea3a7dc37023cd28efa3316f24a26413cbedf4908e018b3576bd04e9905d3"}
Nov 02 20:54:29.158473 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:29.158412    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" podStartSLOduration=2.158397126 podStartE2EDuration="2.158397126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:29.158111883 +0000 UTC m=+3950.155800488" watchObservedRunningTime="2026-11-02 20:54:29.158397126 +0000 UTC m=+3950.156085734"
Nov 02 20:54:29.217244 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:29.217210    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:29.218829 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:29.218795    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 20:54:29.218948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:29.218857    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 20:54:29.591618 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:29.573644    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:29.796790 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:29.796468    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:30.217258 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:30.217224    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 20:54:30.217438 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:30.217285    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 20:54:30.974874 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:30.974838    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:31.217047 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:31.217014    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 20:54:31.217224 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:31.217063    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 20:54:32.184339 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:32.184304    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:32.216977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:32.216948    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 20:54:32.217156 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:32.216993    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 20:54:32.399579 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:32.399537    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:33.193448 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:33.193416    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"]
Nov 02 20:54:33.216884 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:33.216857    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 20:54:33.217039 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:33.216907    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 20:54:33.379247 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:33.379210    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"]
Nov 02 20:54:33.617418 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:33.617366    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:33.838261 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:33.838229    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:34.016665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:34.016618    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:34.203036 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:34.202997    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:34.217097 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:34.217066    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 20:54:34.217242 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:34.217118    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 20:54:35.216982 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:35.216944    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 20:54:35.217367 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:35.217006    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 20:54:36.217651 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:36.217618    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 20:54:36.218161 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:36.217674    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 20:54:36.365504 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:36.365468    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:37.308000 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:37.307955    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:37.308600 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:37.308442    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:37.309567 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:37.309544    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 20:54:38.194414 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.194378    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4" podUID="9e4e5727-61d0-472f-b24f-d264e21e4783" containerName="payload-processing" containerID="cri-o://7b967f1e31d45d5357c4be6bb889bc62d1a9f32b1d298a5f467a056b934b08e5" gracePeriod=25
Nov 02 20:54:38.379693 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.379646    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz" podUID="89904cd2-94b1-4b5a-b7ee-f070a316ec4e" containerName="payload-pre-processing" containerID="cri-o://06a9734293de106e4ccf09e3e30ed5154d67655155f43771b3b2af49ba6c2f52" gracePeriod=25
Nov 02 20:54:38.511017 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.510991    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"
Nov 02 20:54:38.594821 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.594789    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e4e5727-61d0-472f-b24f-d264e21e4783-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4e5727-61d0-472f-b24f-d264e21e4783-plugins-config-volume\") pod \"9e4e5727-61d0-472f-b24f-d264e21e4783\" (UID: \"9e4e5727-61d0-472f-b24f-d264e21e4783\") "
Nov 02 20:54:38.594983 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.594906    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e4e5727-61d0-472f-b24f-d264e21e4783-kube-api-access-ktnsm\" (UniqueName: \"kubernetes.io/projected/9e4e5727-61d0-472f-b24f-d264e21e4783-kube-api-access-ktnsm\") pod \"9e4e5727-61d0-472f-b24f-d264e21e4783\" (UID: \"9e4e5727-61d0-472f-b24f-d264e21e4783\") "
Nov 02 20:54:38.595170 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.595138    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4e5727-61d0-472f-b24f-d264e21e4783-plugins-config-volume" pod "9e4e5727-61d0-472f-b24f-d264e21e4783" (UID: "9e4e5727-61d0-472f-b24f-d264e21e4783"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:54:38.595301 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.595283    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4e5727-61d0-472f-b24f-d264e21e4783-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:54:38.597579 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.597554    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4e5727-61d0-472f-b24f-d264e21e4783-kube-api-access-ktnsm" pod "9e4e5727-61d0-472f-b24f-d264e21e4783" (UID: "9e4e5727-61d0-472f-b24f-d264e21e4783"). InnerVolumeSpecName "kube-api-access-ktnsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:54:38.642161 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.642138    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"
Nov 02 20:54:38.695858 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.695821    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-kube-api-access-z5wqz\" (UniqueName: \"kubernetes.io/projected/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-kube-api-access-z5wqz\") pod \"89904cd2-94b1-4b5a-b7ee-f070a316ec4e\" (UID: \"89904cd2-94b1-4b5a-b7ee-f070a316ec4e\") "
Nov 02 20:54:38.696065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.695936    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-plugins-config-volume\") pod \"89904cd2-94b1-4b5a-b7ee-f070a316ec4e\" (UID: \"89904cd2-94b1-4b5a-b7ee-f070a316ec4e\") "
Nov 02 20:54:38.696307 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.696284    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-plugins-config-volume" pod "89904cd2-94b1-4b5a-b7ee-f070a316ec4e" (UID: "89904cd2-94b1-4b5a-b7ee-f070a316ec4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:54:38.696398 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.696366    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktnsm\" (UniqueName: \"kubernetes.io/projected/9e4e5727-61d0-472f-b24f-d264e21e4783-kube-api-access-ktnsm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:54:38.697993 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.697973    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-kube-api-access-z5wqz" pod "89904cd2-94b1-4b5a-b7ee-f070a316ec4e" (UID: "89904cd2-94b1-4b5a-b7ee-f070a316ec4e"). InnerVolumeSpecName "kube-api-access-z5wqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:54:38.797734 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.797698    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5wqz\" (UniqueName: \"kubernetes.io/projected/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-kube-api-access-z5wqz\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:54:38.797734 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:38.797726    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89904cd2-94b1-4b5a-b7ee-f070a316ec4e-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:54:39.061282 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.061199    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:39.155864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.152684    5224 generic.go:356] "Generic (PLEG): container finished" podID="89904cd2-94b1-4b5a-b7ee-f070a316ec4e" containerID="06a9734293de106e4ccf09e3e30ed5154d67655155f43771b3b2af49ba6c2f52" exitCode=0
Nov 02 20:54:39.155864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.152815    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz" event={"ID":"89904cd2-94b1-4b5a-b7ee-f070a316ec4e","Type":"ContainerDied","Data":"06a9734293de106e4ccf09e3e30ed5154d67655155f43771b3b2af49ba6c2f52"}
Nov 02 20:54:39.155864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.152866    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz" event={"ID":"89904cd2-94b1-4b5a-b7ee-f070a316ec4e","Type":"ContainerDied","Data":"bc17ed950c173fd07aea83a367f7ce0077e718e941c00c580c07d1cd71a512aa"}
Nov 02 20:54:39.155864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.152895    5224 scope.go:122] "RemoveContainer" containerID="06a9734293de106e4ccf09e3e30ed5154d67655155f43771b3b2af49ba6c2f52"
Nov 02 20:54:39.155864 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.153118    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"
Nov 02 20:54:39.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.166346    5224 generic.go:356] "Generic (PLEG): container finished" podID="9e4e5727-61d0-472f-b24f-d264e21e4783" containerID="7b967f1e31d45d5357c4be6bb889bc62d1a9f32b1d298a5f467a056b934b08e5" exitCode=0
Nov 02 20:54:39.167903 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.167255    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"
Nov 02 20:54:39.171663 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.171559    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4" event={"ID":"9e4e5727-61d0-472f-b24f-d264e21e4783","Type":"ContainerDied","Data":"7b967f1e31d45d5357c4be6bb889bc62d1a9f32b1d298a5f467a056b934b08e5"}
Nov 02 20:54:39.173313 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.173287    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4" event={"ID":"9e4e5727-61d0-472f-b24f-d264e21e4783","Type":"ContainerDied","Data":"83d00a48fbc1c482e20fedab0f016f0b0cd0e57b6219201907070cfcc832cf0b"}
Nov 02 20:54:39.178739 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.178716    5224 scope.go:122] "RemoveContainer" containerID="06a9734293de106e4ccf09e3e30ed5154d67655155f43771b3b2af49ba6c2f52"
Nov 02 20:54:39.179170 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:54:39.179133    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a9734293de106e4ccf09e3e30ed5154d67655155f43771b3b2af49ba6c2f52\": container with ID starting with 06a9734293de106e4ccf09e3e30ed5154d67655155f43771b3b2af49ba6c2f52 not found: ID does not exist" containerID="06a9734293de106e4ccf09e3e30ed5154d67655155f43771b3b2af49ba6c2f52"
Nov 02 20:54:39.179265 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.179177    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a9734293de106e4ccf09e3e30ed5154d67655155f43771b3b2af49ba6c2f52"} err="failed to get container status \"06a9734293de106e4ccf09e3e30ed5154d67655155f43771b3b2af49ba6c2f52\": rpc error: code = NotFound desc = could not find container \"06a9734293de106e4ccf09e3e30ed5154d67655155f43771b3b2af49ba6c2f52\": container with ID starting with 06a9734293de106e4ccf09e3e30ed5154d67655155f43771b3b2af49ba6c2f52 not found: ID does not exist"
Nov 02 20:54:39.179265 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.179195    5224 scope.go:122] "RemoveContainer" containerID="7b967f1e31d45d5357c4be6bb889bc62d1a9f32b1d298a5f467a056b934b08e5"
Nov 02 20:54:39.189099 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.189078    5224 scope.go:122] "RemoveContainer" containerID="7b967f1e31d45d5357c4be6bb889bc62d1a9f32b1d298a5f467a056b934b08e5"
Nov 02 20:54:39.189355 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:54:39.189334    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b967f1e31d45d5357c4be6bb889bc62d1a9f32b1d298a5f467a056b934b08e5\": container with ID starting with 7b967f1e31d45d5357c4be6bb889bc62d1a9f32b1d298a5f467a056b934b08e5 not found: ID does not exist" containerID="7b967f1e31d45d5357c4be6bb889bc62d1a9f32b1d298a5f467a056b934b08e5"
Nov 02 20:54:39.189408 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.189363    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b967f1e31d45d5357c4be6bb889bc62d1a9f32b1d298a5f467a056b934b08e5"} err="failed to get container status \"7b967f1e31d45d5357c4be6bb889bc62d1a9f32b1d298a5f467a056b934b08e5\": rpc error: code = NotFound desc = could not find container \"7b967f1e31d45d5357c4be6bb889bc62d1a9f32b1d298a5f467a056b934b08e5\": container with ID starting with 7b967f1e31d45d5357c4be6bb889bc62d1a9f32b1d298a5f467a056b934b08e5 not found: ID does not exist"
Nov 02 20:54:39.240474 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.240449    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:39.328141 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.328060    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"]
Nov 02 20:54:39.330329 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.330306    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-020389-5c74b7589-2zld4"]
Nov 02 20:54:39.439752 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.439719    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"]
Nov 02 20:54:39.441869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.441847    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-020389-c5f7b65c5-5vpsz"]
Nov 02 20:54:39.856681 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:39.856644    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:41.168153 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:41.168114    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89904cd2-94b1-4b5a-b7ee-f070a316ec4e" path="/var/lib/kubelet/pods/89904cd2-94b1-4b5a-b7ee-f070a316ec4e/volumes"
Nov 02 20:54:41.168554 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:41.168509    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e4e5727-61d0-472f-b24f-d264e21e4783" path="/var/lib/kubelet/pods/9e4e5727-61d0-472f-b24f-d264e21e4783/volumes"
Nov 02 20:54:41.746024 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:41.745963    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:43.203420 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:43.203371    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:43.375687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:43.375653    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:46.531515 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:46.531475    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"]
Nov 02 20:54:46.532011 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:46.531724    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v" podUID="4351f7c6-66c0-447b-bd89-e2d35d3d43b7" containerName="maas-api" containerID="cri-o://a2808f16d1fb32be8ed4bb797a3a6a7b89a289e39fc2c4aec473b78a8a912b4e" gracePeriod=30
Nov 02 20:54:46.731685 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:46.731650    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:46.837835 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:46.837808    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:54:46.905827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:46.905783    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-kube-api-access-cx8wj\" (UniqueName: \"kubernetes.io/projected/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-kube-api-access-cx8wj\") pod \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") "
Nov 02 20:54:46.906029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:46.905844    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-metrics-certs\") pod \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") "
Nov 02 20:54:46.906029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:46.905888    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-openshift-service-ca\") pod \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") "
Nov 02 20:54:46.906029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:46.905915    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-maas-api-tls\") pod \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\" (UID: \"4351f7c6-66c0-447b-bd89-e2d35d3d43b7\") "
Nov 02 20:54:46.906557 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:46.906516    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-openshift-service-ca" pod "4351f7c6-66c0-447b-bd89-e2d35d3d43b7" (UID: "4351f7c6-66c0-447b-bd89-e2d35d3d43b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:54:46.908229 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:46.908203    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-metrics-certs" pod "4351f7c6-66c0-447b-bd89-e2d35d3d43b7" (UID: "4351f7c6-66c0-447b-bd89-e2d35d3d43b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:54:46.908350 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:46.908264    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-maas-api-tls" pod "4351f7c6-66c0-447b-bd89-e2d35d3d43b7" (UID: "4351f7c6-66c0-447b-bd89-e2d35d3d43b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:54:46.908428 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:46.908385    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-kube-api-access-cx8wj" pod "4351f7c6-66c0-447b-bd89-e2d35d3d43b7" (UID: "4351f7c6-66c0-447b-bd89-e2d35d3d43b7"). InnerVolumeSpecName "kube-api-access-cx8wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:54:47.008176 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:47.008135    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx8wj\" (UniqueName: \"kubernetes.io/projected/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-kube-api-access-cx8wj\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:54:47.008176 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:47.008177    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:54:47.008392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:47.008194    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:54:47.008392 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:47.008209    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4351f7c6-66c0-447b-bd89-e2d35d3d43b7-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:54:47.223293 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:47.223263    5224 generic.go:356] "Generic (PLEG): container finished" podID="4351f7c6-66c0-447b-bd89-e2d35d3d43b7" containerID="a2808f16d1fb32be8ed4bb797a3a6a7b89a289e39fc2c4aec473b78a8a912b4e" exitCode=0
Nov 02 20:54:47.223497 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:47.223310    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"
Nov 02 20:54:47.223497 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:47.223351    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v" event={"ID":"4351f7c6-66c0-447b-bd89-e2d35d3d43b7","Type":"ContainerDied","Data":"a2808f16d1fb32be8ed4bb797a3a6a7b89a289e39fc2c4aec473b78a8a912b4e"}
Nov 02 20:54:47.223497 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:47.223409    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v" event={"ID":"4351f7c6-66c0-447b-bd89-e2d35d3d43b7","Type":"ContainerDied","Data":"fcddc16a7276c7fae86a76c7fa7fe316187b6e696010f22a5155ed0291d73341"}
Nov 02 20:54:47.223497 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:47.223439    5224 scope.go:122] "RemoveContainer" containerID="a2808f16d1fb32be8ed4bb797a3a6a7b89a289e39fc2c4aec473b78a8a912b4e"
Nov 02 20:54:47.232500 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:47.232483    5224 scope.go:122] "RemoveContainer" containerID="a2808f16d1fb32be8ed4bb797a3a6a7b89a289e39fc2c4aec473b78a8a912b4e"
Nov 02 20:54:47.232813 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:54:47.232795    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2808f16d1fb32be8ed4bb797a3a6a7b89a289e39fc2c4aec473b78a8a912b4e\": container with ID starting with a2808f16d1fb32be8ed4bb797a3a6a7b89a289e39fc2c4aec473b78a8a912b4e not found: ID does not exist" containerID="a2808f16d1fb32be8ed4bb797a3a6a7b89a289e39fc2c4aec473b78a8a912b4e"
Nov 02 20:54:47.232869 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:47.232823    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2808f16d1fb32be8ed4bb797a3a6a7b89a289e39fc2c4aec473b78a8a912b4e"} err="failed to get container status \"a2808f16d1fb32be8ed4bb797a3a6a7b89a289e39fc2c4aec473b78a8a912b4e\": rpc error: code = NotFound desc = could not find container \"a2808f16d1fb32be8ed4bb797a3a6a7b89a289e39fc2c4aec473b78a8a912b4e\": container with ID starting with a2808f16d1fb32be8ed4bb797a3a6a7b89a289e39fc2c4aec473b78a8a912b4e not found: ID does not exist"
Nov 02 20:54:47.394119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:47.394080    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"]
Nov 02 20:54:47.397234 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:47.397212    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-28197c-8695fc5566-49k6v"]
Nov 02 20:54:48.863980 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:48.863887    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:49.167463 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:49.167383    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4351f7c6-66c0-447b-bd89-e2d35d3d43b7" path="/var/lib/kubelet/pods/4351f7c6-66c0-447b-bd89-e2d35d3d43b7/volumes"
Nov 02 20:54:53.944897 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:53.944855    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:55.010897 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:55.010867    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:55.188248 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:55.188209    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:56.572717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:56.572677    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:57.002383 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:57.002293    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"]
Nov 02 20:54:57.002564 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:57.002541    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7" podUID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerName="istio-proxy" containerID="cri-o://aaa59d408159bf0844eefb6200f13cc63761a7cca30b7985f1cc8337525d8ca6" gracePeriod=30
Nov 02 20:54:57.044089 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:57.044048    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:57.359731 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:54:57.359691    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:54:59.053966 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:59.053934    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:54:59.362881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:54:59.362803    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:00.358624 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:00.358568    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:00.635231 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:00.635139    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:02.278070 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.278006    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:55:02.311430 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.311401    5224 generic.go:356] "Generic (PLEG): container finished" podID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerID="aaa59d408159bf0844eefb6200f13cc63761a7cca30b7985f1cc8337525d8ca6" exitCode=0
Nov 02 20:55:02.311598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.311462    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"
Nov 02 20:55:02.311598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.311491    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7" event={"ID":"67dd13d0-2c6c-4838-ba70-dc6053151dc7","Type":"ContainerDied","Data":"aaa59d408159bf0844eefb6200f13cc63761a7cca30b7985f1cc8337525d8ca6"}
Nov 02 20:55:02.311598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.311543    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7" event={"ID":"67dd13d0-2c6c-4838-ba70-dc6053151dc7","Type":"ContainerDied","Data":"51c2f944a179a8ea7f723fd6e66efb603e2c26ea65763bc5032d4132b9d96ec4"}
Nov 02 20:55:02.311598 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.311566    5224 scope.go:122] "RemoveContainer" containerID="aaa59d408159bf0844eefb6200f13cc63761a7cca30b7985f1cc8337525d8ca6"
Nov 02 20:55:02.321020 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.321002    5224 scope.go:122] "RemoveContainer" containerID="fe99b22f92154447f569cd4bde2919973652fba5e47028f64f0cf5c8a21ced36"
Nov 02 20:55:02.329445 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.329427    5224 scope.go:122] "RemoveContainer" containerID="aaa59d408159bf0844eefb6200f13cc63761a7cca30b7985f1cc8337525d8ca6"
Nov 02 20:55:02.329702 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:55:02.329683    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa59d408159bf0844eefb6200f13cc63761a7cca30b7985f1cc8337525d8ca6\": container with ID starting with aaa59d408159bf0844eefb6200f13cc63761a7cca30b7985f1cc8337525d8ca6 not found: ID does not exist" containerID="aaa59d408159bf0844eefb6200f13cc63761a7cca30b7985f1cc8337525d8ca6"
Nov 02 20:55:02.329761 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.329710    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa59d408159bf0844eefb6200f13cc63761a7cca30b7985f1cc8337525d8ca6"} err="failed to get container status \"aaa59d408159bf0844eefb6200f13cc63761a7cca30b7985f1cc8337525d8ca6\": rpc error: code = NotFound desc = could not find container \"aaa59d408159bf0844eefb6200f13cc63761a7cca30b7985f1cc8337525d8ca6\": container with ID starting with aaa59d408159bf0844eefb6200f13cc63761a7cca30b7985f1cc8337525d8ca6 not found: ID does not exist"
Nov 02 20:55:02.329761 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.329726    5224 scope.go:122] "RemoveContainer" containerID="fe99b22f92154447f569cd4bde2919973652fba5e47028f64f0cf5c8a21ced36"
Nov 02 20:55:02.329986 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:55:02.329965    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe99b22f92154447f569cd4bde2919973652fba5e47028f64f0cf5c8a21ced36\": container with ID starting with fe99b22f92154447f569cd4bde2919973652fba5e47028f64f0cf5c8a21ced36 not found: ID does not exist" containerID="fe99b22f92154447f569cd4bde2919973652fba5e47028f64f0cf5c8a21ced36"
Nov 02 20:55:02.330046 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.329995    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe99b22f92154447f569cd4bde2919973652fba5e47028f64f0cf5c8a21ced36"} err="failed to get container status \"fe99b22f92154447f569cd4bde2919973652fba5e47028f64f0cf5c8a21ced36\": rpc error: code = NotFound desc = could not find container \"fe99b22f92154447f569cd4bde2919973652fba5e47028f64f0cf5c8a21ced36\": container with ID starting with fe99b22f92154447f569cd4bde2919973652fba5e47028f64f0cf5c8a21ced36 not found: ID does not exist"
Nov 02 20:55:02.379493 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.379470    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-certs\") pod \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") "
Nov 02 20:55:02.379608 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.379532    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-podinfo\") pod \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") "
Nov 02 20:55:02.379608 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.379561    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-kube-api-access-blftx\" (UniqueName: \"kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-kube-api-access-blftx\") pod \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") "
Nov 02 20:55:02.379608 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.379585    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-envoy\") pod \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") "
Nov 02 20:55:02.379758 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.379645    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-data\") pod \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") "
Nov 02 20:55:02.379758 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.379699    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-credential-socket\") pod \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") "
Nov 02 20:55:02.379924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.379749    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-certs" pod "67dd13d0-2c6c-4838-ba70-dc6053151dc7" (UID: "67dd13d0-2c6c-4838-ba70-dc6053151dc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:55:02.379924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.379859    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istiod-ca-cert\") pod \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") "
Nov 02 20:55:02.379924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.379917    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-token\" (UniqueName: \"kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-token\") pod \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") "
Nov 02 20:55:02.380110 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.379988    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-socket\") pod \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\" (UID: \"67dd13d0-2c6c-4838-ba70-dc6053151dc7\") "
Nov 02 20:55:02.380175 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.380132    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-credential-socket" pod "67dd13d0-2c6c-4838-ba70-dc6053151dc7" (UID: "67dd13d0-2c6c-4838-ba70-dc6053151dc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:55:02.380230 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.380213    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istiod-ca-cert" pod "67dd13d0-2c6c-4838-ba70-dc6053151dc7" (UID: "67dd13d0-2c6c-4838-ba70-dc6053151dc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:55:02.380321 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.380297    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-data" pod "67dd13d0-2c6c-4838-ba70-dc6053151dc7" (UID: "67dd13d0-2c6c-4838-ba70-dc6053151dc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:55:02.380449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.380415    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-socket" pod "67dd13d0-2c6c-4838-ba70-dc6053151dc7" (UID: "67dd13d0-2c6c-4838-ba70-dc6053151dc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:55:02.380687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.380614    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:02.380687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.380637    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:02.380687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.380653    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:02.380687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.380663    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:02.380687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.380672    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:02.381929 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.381902    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-podinfo" pod "67dd13d0-2c6c-4838-ba70-dc6053151dc7" (UID: "67dd13d0-2c6c-4838-ba70-dc6053151dc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:55:02.382154 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.382134    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-envoy" pod "67dd13d0-2c6c-4838-ba70-dc6053151dc7" (UID: "67dd13d0-2c6c-4838-ba70-dc6053151dc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:55:02.382488 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.382469    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-token" pod "67dd13d0-2c6c-4838-ba70-dc6053151dc7" (UID: "67dd13d0-2c6c-4838-ba70-dc6053151dc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:55:02.382552 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.382486    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-kube-api-access-blftx" pod "67dd13d0-2c6c-4838-ba70-dc6053151dc7" (UID: "67dd13d0-2c6c-4838-ba70-dc6053151dc7"). InnerVolumeSpecName "kube-api-access-blftx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:55:02.482082 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.482049    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:02.482082 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.482081    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blftx\" (UniqueName: \"kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-kube-api-access-blftx\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:02.482281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.482091    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:02.482281 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.482100    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67dd13d0-2c6c-4838-ba70-dc6053151dc7-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:02.733883 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.733850    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"]
Nov 02 20:55:02.737302 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:02.737279    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-28197c-openshift-default-6778f9b7df-cqqj7"]
Nov 02 20:55:03.167667 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:03.167632    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" path="/var/lib/kubelet/pods/67dd13d0-2c6c-4838-ba70-dc6053151dc7/volumes"
Nov 02 20:55:04.405039 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:04.404989    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:04.632464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:04.632430    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:04.827369 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:04.827323    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:05.008370 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:05.008332    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:07.063396 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:07.063353    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:07.790410 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:07.790373    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:07.972532 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:07.972497    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:09.687555 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:09.687522    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:13.214395 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.214359    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"]
Nov 02 20:55:13.214891 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.214876    5224 cpu_manager.go:414] "containerMap: removing container" podUID="9e4e5727-61d0-472f-b24f-d264e21e4783" containerName="payload-processing"
Nov 02 20:55:13.214947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.214892    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4e5727-61d0-472f-b24f-d264e21e4783" containerName="payload-processing"
Nov 02 20:55:13.214947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.214911    5224 cpu_manager.go:414] "containerMap: removing container" podUID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerName="istio-proxy"
Nov 02 20:55:13.214947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.214916    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerName="istio-proxy"
Nov 02 20:55:13.214947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.214926    5224 cpu_manager.go:414] "containerMap: removing container" podUID="89904cd2-94b1-4b5a-b7ee-f070a316ec4e" containerName="payload-pre-processing"
Nov 02 20:55:13.214947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.214931    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89904cd2-94b1-4b5a-b7ee-f070a316ec4e" containerName="payload-pre-processing"
Nov 02 20:55:13.214947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.214939    5224 cpu_manager.go:414] "containerMap: removing container" podUID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerName="istio-proxy"
Nov 02 20:55:13.214947 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.214944    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerName="istio-proxy"
Nov 02 20:55:13.215226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.214955    5224 cpu_manager.go:414] "containerMap: removing container" podUID="4351f7c6-66c0-447b-bd89-e2d35d3d43b7" containerName="maas-api"
Nov 02 20:55:13.215226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.214960    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4351f7c6-66c0-447b-bd89-e2d35d3d43b7" containerName="maas-api"
Nov 02 20:55:13.215226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.215027    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerName="istio-proxy"
Nov 02 20:55:13.215226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.215035    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="89904cd2-94b1-4b5a-b7ee-f070a316ec4e" containerName="payload-pre-processing"
Nov 02 20:55:13.215226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.215042    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="67dd13d0-2c6c-4838-ba70-dc6053151dc7" containerName="istio-proxy"
Nov 02 20:55:13.215226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.215050    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="4351f7c6-66c0-447b-bd89-e2d35d3d43b7" containerName="maas-api"
Nov 02 20:55:13.215226 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.215062    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4e5727-61d0-472f-b24f-d264e21e4783" containerName="payload-processing"
Nov 02 20:55:13.219973 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.219954    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.310624 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.310590    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7456949c-c902-4940-b3e4-64fab197654b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.310624 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.310631    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.310921 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.310656    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.310921 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.310748    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xlv7\" (UniqueName: \"kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-kube-api-access-6xlv7\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.310921 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.310818    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.310921 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.310864    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.310921 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.310907    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.311098 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.310977    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7456949c-c902-4940-b3e4-64fab197654b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.311098 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.311014    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.313948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.313906    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"]
Nov 02 20:55:13.412264 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.412225    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.412264 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.412263    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.412545 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.412287    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xlv7\" (UniqueName: \"kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-kube-api-access-6xlv7\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.412545 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.412322    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.412545 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.412358    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.412545 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.412375    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.412545 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.412422    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7456949c-c902-4940-b3e4-64fab197654b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.412545 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.412497    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.412911 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.412657    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7456949c-c902-4940-b3e4-64fab197654b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.413477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.413448    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.415563 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.415513    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.415868 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.415845    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.416464 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.416416    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7456949c-c902-4940-b3e4-64fab197654b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.416565 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.416526    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7456949c-c902-4940-b3e4-64fab197654b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.418457 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.418435    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.418875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.418851    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.471267 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.471165    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.471445 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.471413    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xlv7\" (UniqueName: \"kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-kube-api-access-6xlv7\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d46vl\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.533576 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.533536    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:13.952873 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.952839    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"]
Nov 02 20:55:13.986202 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:55:13.986162    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7456949c_c902_4940_b3e4_64fab197654b.slice/crio-e8672731b390848500b3bee1a5048f2b1726f4f285b950089b1c33eb19cb46a8 WatchSource:0}: Error finding container e8672731b390848500b3bee1a5048f2b1726f4f285b950089b1c33eb19cb46a8: Status 404 returned error can't find the container with id e8672731b390848500b3bee1a5048f2b1726f4f285b950089b1c33eb19cb46a8
Nov 02 20:55:13.988724 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.988694    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:55:13.988851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.988761    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:55:13.988851 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:13.988811    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:55:14.401152 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:14.401113    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" event={"ID":"7456949c-c902-4940-b3e4-64fab197654b","Type":"ContainerStarted","Data":"b16b9f23d4631b84f9d81e750f1f82ee57b8c62d3f87415048b057fdd259cc09"}
Nov 02 20:55:14.401675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:14.401179    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" event={"ID":"7456949c-c902-4940-b3e4-64fab197654b","Type":"ContainerStarted","Data":"e8672731b390848500b3bee1a5048f2b1726f4f285b950089b1c33eb19cb46a8"}
Nov 02 20:55:14.521849 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:14.521794    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" podStartSLOduration=1.52174803 podStartE2EDuration="1.52174803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:55:14.519314485 +0000 UTC m=+3995.517003093" watchObservedRunningTime="2026-11-02 20:55:14.52174803 +0000 UTC m=+3995.519436638"
Nov 02 20:55:14.533660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:14.533617    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:14.885010 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:14.884970    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:15.535746 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:15.535709    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d46vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 02 20:55:15.536173 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:15.535801    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" podUID="7456949c-c902-4940-b3e4-64fab197654b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 02 20:55:16.534779 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:16.534738    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d46vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 02 20:55:16.534948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:16.534858    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" podUID="7456949c-c902-4940-b3e4-64fab197654b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 02 20:55:17.534952 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:17.534911    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d46vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 02 20:55:17.535359 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:17.534992    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" podUID="7456949c-c902-4940-b3e4-64fab197654b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 02 20:55:18.534196 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:18.534157    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d46vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 02 20:55:18.534400 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:18.534214    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" podUID="7456949c-c902-4940-b3e4-64fab197654b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 02 20:55:19.534347 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:19.534307    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d46vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 02 20:55:19.534808 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:19.534366    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" podUID="7456949c-c902-4940-b3e4-64fab197654b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 02 20:55:19.715595 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:19.715556    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:20.534248 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:20.534204    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d46vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 02 20:55:20.534455 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:20.534266    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" podUID="7456949c-c902-4940-b3e4-64fab197654b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 02 20:55:21.534135 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:21.534097    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d46vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 02 20:55:21.534320 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:21.534153    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" podUID="7456949c-c902-4940-b3e4-64fab197654b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 02 20:55:22.809958 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:22.809918    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:22.810624 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:22.810418    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:22.811444 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:22.811423    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 20:55:24.552115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:24.552044    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:24.730100 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:24.730062    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:55:27.359063 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:55:27.359025    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:55:29.370641 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:29.370601    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:34.267452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:34.267415    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:39.167152 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:39.167107    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:43.647757 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.647722    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"]
Nov 02 20:55:43.652948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.652920    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:43.707340 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.707305    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"]
Nov 02 20:55:43.756031 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.755994    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f97409-serving-cert\""
Nov 02 20:55:43.793350 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.793306    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972297c3-0639-4570-8636-5d29249c866f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:43.793560 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.793428    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-maas-api-tls\") pod \"maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:43.793647 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.793606    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-metrics-certs\") pod \"maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:43.793715 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.793649    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5ch\" (UniqueName: \"kubernetes.io/projected/972297c3-0639-4570-8636-5d29249c866f-kube-api-access-sw5ch\") pod \"maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:43.889917 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.889884    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:43.894959 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.894924    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-metrics-certs\") pod \"maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:43.895138 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.894995    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5ch\" (UniqueName: \"kubernetes.io/projected/972297c3-0639-4570-8636-5d29249c866f-kube-api-access-sw5ch\") pod \"maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:43.895138 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.895059    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972297c3-0639-4570-8636-5d29249c866f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:43.895255 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.895152    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-maas-api-tls\") pod \"maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:43.895333 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:55:43.895314    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-serving-cert: secret "maas-api-e2e-inf-a-f97409-serving-cert" not found
Nov 02 20:55:43.895407 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:55:43.895382    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-maas-api-tls podName:972297c3-0639-4570-8636-5d29249c866f nodeName:}" failed. No retries permitted until 2026-11-02 20:55:44.395363876 +0000 UTC m=+4025.393052479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-maas-api-tls") pod "maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs" (UID: "972297c3-0639-4570-8636-5d29249c866f") : secret "maas-api-e2e-inf-a-f97409-serving-cert" not found
Nov 02 20:55:43.895884 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.895859    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972297c3-0639-4570-8636-5d29249c866f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:43.897976 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.897901    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-metrics-certs\") pod \"maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:43.956194 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:43.956164    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5ch\" (UniqueName: \"kubernetes.io/projected/972297c3-0639-4570-8636-5d29249c866f-kube-api-access-sw5ch\") pod \"maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:44.066301 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:44.066257    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:44.243993 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:44.243892    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:44.401253 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:44.401215    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-maas-api-tls\") pod \"maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:44.403824 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:44.403787    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-maas-api-tls\") pod \"maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:44.568147 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:44.568103    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:44.944677 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:44.944650    5224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:55:44.946828 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:44.946808    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"]
Nov 02 20:55:45.593738 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:45.593700    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs" event={"ID":"972297c3-0639-4570-8636-5d29249c866f","Type":"ContainerStarted","Data":"780dde27af28c8f4ca9ee77a09f374d34f140e6e9e192c61977c4b5ce134af91"}
Nov 02 20:55:46.600393 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:46.600356    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs" event={"ID":"972297c3-0639-4570-8636-5d29249c866f","Type":"ContainerStarted","Data":"234a879d1e4f112b955a10629cf8e6875bc21bc4b686b41148f68b0e9ca45e67"}
Nov 02 20:55:46.600837 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:46.600480    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:46.721055 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:46.721002    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs" podStartSLOduration=2.850801895 podStartE2EDuration="3.720987092s" totalImagesPullingTime="870.185197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:55:46.718429368 +0000 UTC m=+4027.716117978" watchObservedRunningTime="2026-11-02 20:55:46.720987092 +0000 UTC m=+4027.718675701"
Nov 02 20:55:49.000571 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:49.000503    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:52.612547 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:52.612511    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 20:55:53.304111 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:53.304065    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:55.006928 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:55.006889    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:55.192308 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:55.192272    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:55.363735 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:55.363696    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:57.365535 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:55:57.365495    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:55:58.075193 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:58.075149    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:58.524590 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:58.524549    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"]
Nov 02 20:55:58.525053 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:58.524762    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerName="istio-proxy" containerID="cri-o://9f808f76869abecee8e792be1912899e1c1f8c532b5281048c3dc63b4a291290" gracePeriod=30
Nov 02 20:55:58.567228 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:58.567192    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:55:59.388295 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:55:59.388258    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:00.367398 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:00.367361    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:00.618573 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:00.618484    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:00.822322 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:00.822284    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:03.062535 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:03.062447    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:03.243630 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:03.243593    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:03.730137 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:03.730094    5224 generic.go:356] "Generic (PLEG): container finished" podID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerID="9f808f76869abecee8e792be1912899e1c1f8c532b5281048c3dc63b4a291290" exitCode=0
Nov 02 20:56:03.730343 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:03.730270    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" event={"ID":"d2424b57-879c-47f4-b608-6e7731fcefaf","Type":"ContainerDied","Data":"9f808f76869abecee8e792be1912899e1c1f8c532b5281048c3dc63b4a291290"}
Nov 02 20:56:03.730343 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:03.730337    5224 scope.go:122] "RemoveContainer" containerID="d629c6e5981416881b80483bacf7daffaf2b3e863f568f8a6019e4629a3e8657"
Nov 02 20:56:04.072810 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.072113    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:56:04.148031 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.147937    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-podinfo\") pod \"d2424b57-879c-47f4-b608-6e7731fcefaf\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") "
Nov 02 20:56:04.148203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148041    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-certs\") pod \"d2424b57-879c-47f4-b608-6e7731fcefaf\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") "
Nov 02 20:56:04.148203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148117    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-token\" (UniqueName: \"kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-token\") pod \"d2424b57-879c-47f4-b608-6e7731fcefaf\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") "
Nov 02 20:56:04.148203 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148177    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-data\") pod \"d2424b57-879c-47f4-b608-6e7731fcefaf\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") "
Nov 02 20:56:04.148376 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148267    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-credential-socket\") pod \"d2424b57-879c-47f4-b608-6e7731fcefaf\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") "
Nov 02 20:56:04.148376 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148314    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2424b57-879c-47f4-b608-6e7731fcefaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2424b57-879c-47f4-b608-6e7731fcefaf-istiod-ca-cert\") pod \"d2424b57-879c-47f4-b608-6e7731fcefaf\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") "
Nov 02 20:56:04.148376 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148353    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-kube-api-access-dg4qf\" (UniqueName: \"kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-kube-api-access-dg4qf\") pod \"d2424b57-879c-47f4-b608-6e7731fcefaf\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") "
Nov 02 20:56:04.148376 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148352    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-certs" pod "d2424b57-879c-47f4-b608-6e7731fcefaf" (UID: "d2424b57-879c-47f4-b608-6e7731fcefaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:56:04.148607 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148399    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-socket\") pod \"d2424b57-879c-47f4-b608-6e7731fcefaf\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") "
Nov 02 20:56:04.148607 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148448    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-envoy\") pod \"d2424b57-879c-47f4-b608-6e7731fcefaf\" (UID: \"d2424b57-879c-47f4-b608-6e7731fcefaf\") "
Nov 02 20:56:04.148607 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148501    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-data" pod "d2424b57-879c-47f4-b608-6e7731fcefaf" (UID: "d2424b57-879c-47f4-b608-6e7731fcefaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:56:04.148607 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148564    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-credential-socket" pod "d2424b57-879c-47f4-b608-6e7731fcefaf" (UID: "d2424b57-879c-47f4-b608-6e7731fcefaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:56:04.148871 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148840    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2424b57-879c-47f4-b608-6e7731fcefaf-istiod-ca-cert" pod "d2424b57-879c-47f4-b608-6e7731fcefaf" (UID: "d2424b57-879c-47f4-b608-6e7731fcefaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:56:04.150949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148938    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:56:04.150949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148960    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2424b57-879c-47f4-b608-6e7731fcefaf-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:56:04.150949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148975    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:56:04.150949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.148990    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:56:04.150949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.149160    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-socket" pod "d2424b57-879c-47f4-b608-6e7731fcefaf" (UID: "d2424b57-879c-47f4-b608-6e7731fcefaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:56:04.150949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.150700    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-podinfo" pod "d2424b57-879c-47f4-b608-6e7731fcefaf" (UID: "d2424b57-879c-47f4-b608-6e7731fcefaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:56:04.151452 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.151282    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-envoy" pod "d2424b57-879c-47f4-b608-6e7731fcefaf" (UID: "d2424b57-879c-47f4-b608-6e7731fcefaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:56:04.151703 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.151664    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-token" pod "d2424b57-879c-47f4-b608-6e7731fcefaf" (UID: "d2424b57-879c-47f4-b608-6e7731fcefaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:56:04.161115 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.161077    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-kube-api-access-dg4qf" pod "d2424b57-879c-47f4-b608-6e7731fcefaf" (UID: "d2424b57-879c-47f4-b608-6e7731fcefaf"). InnerVolumeSpecName "kube-api-access-dg4qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:56:04.249625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.249593    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:56:04.249625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.249624    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:56:04.249625 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.249634    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:56:04.249897 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.249643    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:56:04.249897 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.249652    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg4qf\" (UniqueName: \"kubernetes.io/projected/d2424b57-879c-47f4-b608-6e7731fcefaf-kube-api-access-dg4qf\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:56:04.736665 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.736636    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"
Nov 02 20:56:04.736900 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.736690    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4" event={"ID":"d2424b57-879c-47f4-b608-6e7731fcefaf","Type":"ContainerDied","Data":"a76bce61b4bf01fc55dd2530c78b2cd98b163c86a507008ebd47818b511666f5"}
Nov 02 20:56:04.736900 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.736748    5224 scope.go:122] "RemoveContainer" containerID="9f808f76869abecee8e792be1912899e1c1f8c532b5281048c3dc63b4a291290"
Nov 02 20:56:04.955564 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.955517    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"]
Nov 02 20:56:04.958954 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:04.958930    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-020389-openshift-default-8f4b9dd5-fx4l4"]
Nov 02 20:56:05.139587 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:05.139545    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:05.167436 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:05.167406    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" path="/var/lib/kubelet/pods/d2424b57-879c-47f4-b608-6e7731fcefaf/volumes"
Nov 02 20:56:05.333348 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:05.333313    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:05.588572 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:05.588538    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:05.798291 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:05.798236    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:07.390449 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:07.390406    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:08.282299 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.282267    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:08.499997 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.499955    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:08.649807 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.649690    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"]
Nov 02 20:56:08.650316 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.650296    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerName="istio-proxy"
Nov 02 20:56:08.650427 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.650317    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerName="istio-proxy"
Nov 02 20:56:08.650427 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.650343    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerName="istio-proxy"
Nov 02 20:56:08.650427 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.650350    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerName="istio-proxy"
Nov 02 20:56:08.650581 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.650495    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerName="istio-proxy"
Nov 02 20:56:08.650581 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.650511    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2424b57-879c-47f4-b608-6e7731fcefaf" containerName="istio-proxy"
Nov 02 20:56:08.655267 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.655247    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"
Nov 02 20:56:08.700229 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.700199    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlb7n\" (UniqueName: \"kubernetes.io/projected/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-kube-api-access-tlb7n\") pod \"payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh\" (UID: \"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"
Nov 02 20:56:08.700409 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.700369    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh\" (UID: \"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"
Nov 02 20:56:08.705448 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.705420    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"]
Nov 02 20:56:08.755729 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.755535    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4530c5\""
Nov 02 20:56:08.755729 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.755695    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4530c5-dockercfg-dchh4\""
Nov 02 20:56:08.801272 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.801236    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh\" (UID: \"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"
Nov 02 20:56:08.801457 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.801333    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlb7n\" (UniqueName: \"kubernetes.io/projected/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-kube-api-access-tlb7n\") pod \"payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh\" (UID: \"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"
Nov 02 20:56:08.802045 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.802026    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh\" (UID: \"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"
Nov 02 20:56:08.809191 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.809163    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"]
Nov 02 20:56:08.813265 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.813244    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"
Nov 02 20:56:08.861599 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.861565    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"]
Nov 02 20:56:08.861863 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.861843    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlb7n\" (UniqueName: \"kubernetes.io/projected/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-kube-api-access-tlb7n\") pod \"payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh\" (UID: \"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"
Nov 02 20:56:08.902519 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.902426    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045ccbb5-32a4-4ccb-8ed7-f17fff958929-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm\" (UID: \"045ccbb5-32a4-4ccb-8ed7-f17fff958929\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"
Nov 02 20:56:08.902699 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.902594    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvn4\" (UniqueName: \"kubernetes.io/projected/045ccbb5-32a4-4ccb-8ed7-f17fff958929-kube-api-access-vtvn4\") pod \"payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm\" (UID: \"045ccbb5-32a4-4ccb-8ed7-f17fff958929\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"
Nov 02 20:56:08.969785 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:08.969736    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"
Nov 02 20:56:09.004177 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.004143    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045ccbb5-32a4-4ccb-8ed7-f17fff958929-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm\" (UID: \"045ccbb5-32a4-4ccb-8ed7-f17fff958929\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"
Nov 02 20:56:09.004332 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.004270    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvn4\" (UniqueName: \"kubernetes.io/projected/045ccbb5-32a4-4ccb-8ed7-f17fff958929-kube-api-access-vtvn4\") pod \"payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm\" (UID: \"045ccbb5-32a4-4ccb-8ed7-f17fff958929\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"
Nov 02 20:56:09.004855 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.004834    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045ccbb5-32a4-4ccb-8ed7-f17fff958929-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm\" (UID: \"045ccbb5-32a4-4ccb-8ed7-f17fff958929\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"
Nov 02 20:56:09.062968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.062914    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvn4\" (UniqueName: \"kubernetes.io/projected/045ccbb5-32a4-4ccb-8ed7-f17fff958929-kube-api-access-vtvn4\") pod \"payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm\" (UID: \"045ccbb5-32a4-4ccb-8ed7-f17fff958929\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"
Nov 02 20:56:09.126833 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.126796    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"
Nov 02 20:56:09.338705 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:56:09.338676    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0b0bcc_fa85_4dad_b946_2d9917a9dce6.slice/crio-f94a9eba03b981f687b0f31cbda6c449fa6f0c8a64b4baf31a6c69b306beae1a WatchSource:0}: Error finding container f94a9eba03b981f687b0f31cbda6c449fa6f0c8a64b4baf31a6c69b306beae1a: Status 404 returned error can't find the container with id f94a9eba03b981f687b0f31cbda6c449fa6f0c8a64b4baf31a6c69b306beae1a
Nov 02 20:56:09.341696 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.341672    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"]
Nov 02 20:56:09.494559 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:56:09.494524    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045ccbb5_32a4_4ccb_8ed7_f17fff958929.slice/crio-d9b031af4e16315e065ea56254f47756138a4e2891bfdc1ebdb5bf2944996569 WatchSource:0}: Error finding container d9b031af4e16315e065ea56254f47756138a4e2891bfdc1ebdb5bf2944996569: Status 404 returned error can't find the container with id d9b031af4e16315e065ea56254f47756138a4e2891bfdc1ebdb5bf2944996569
Nov 02 20:56:09.497759 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.497736    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"]
Nov 02 20:56:09.772505 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.772470    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh" event={"ID":"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6","Type":"ContainerStarted","Data":"8c16cfb5a40dca851c05fb58e2962aef02db9fb4eb81638bbc8be91e6a1d6d8a"}
Nov 02 20:56:09.772969 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.772531    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh" event={"ID":"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6","Type":"ContainerStarted","Data":"f94a9eba03b981f687b0f31cbda6c449fa6f0c8a64b4baf31a6c69b306beae1a"}
Nov 02 20:56:09.772969 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.772573    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"
Nov 02 20:56:09.774083 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.774057    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm" event={"ID":"045ccbb5-32a4-4ccb-8ed7-f17fff958929","Type":"ContainerStarted","Data":"08ed3530bac3b1e0fb177ba8e3f4ca6bed2e4c5ef75d801a303f137f3feb1a6b"}
Nov 02 20:56:09.774192 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.774100    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm" event={"ID":"045ccbb5-32a4-4ccb-8ed7-f17fff958929","Type":"ContainerStarted","Data":"d9b031af4e16315e065ea56254f47756138a4e2891bfdc1ebdb5bf2944996569"}
Nov 02 20:56:09.774192 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.774120    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"
Nov 02 20:56:09.888923 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:09.888865    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh" podStartSLOduration=1.888849022 podStartE2EDuration="1.888849022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:56:09.886860915 +0000 UTC m=+4050.884549524" watchObservedRunningTime="2026-11-02 20:56:09.888849022 +0000 UTC m=+4050.886537630"
Nov 02 20:56:10.003178 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:10.003117    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm" podStartSLOduration=2.00309615 podStartE2EDuration="2.00309615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:56:10.001871748 +0000 UTC m=+4050.999560359" watchObservedRunningTime="2026-11-02 20:56:10.00309615 +0000 UTC m=+4051.000784752"
Nov 02 20:56:11.213552 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:11.213512    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:16.348984 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:16.348948    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:20.783130 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:20.783048    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"
Nov 02 20:56:20.783584 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:20.783246    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"
Nov 02 20:56:21.171707 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:21.171609    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:23.494799 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:23.494750    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:56:25.854849 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:25.854811    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"]
Nov 02 20:56:25.858976 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:25.858958    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:25.912219 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:25.912183    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"]
Nov 02 20:56:25.960695 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:25.960661    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1dd5d5-serving-cert\""
Nov 02 20:56:26.007816 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.007758    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:26.007971 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.007846    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7nd4\" (UniqueName: \"kubernetes.io/projected/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-kube-api-access-j7nd4\") pod \"maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:26.008013 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.007997    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-metrics-certs\") pod \"maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:26.008210 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.008187    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:26.105520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.105376    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:26.109322 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.109287    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:26.109520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.109342    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:26.109520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.109367    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7nd4\" (UniqueName: \"kubernetes.io/projected/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-kube-api-access-j7nd4\") pod \"maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:26.109520 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.109493    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-metrics-certs\") pod \"maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:26.109718 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:56:26.109692    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-serving-cert: secret "maas-api-e2e-inf-a-1dd5d5-serving-cert" not found
Nov 02 20:56:26.109795 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:56:26.109788    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-maas-api-tls podName:71c22cf4-9ac4-4ad9-a59e-27afb7bee488 nodeName:}" failed. No retries permitted until 2026-11-02 20:56:26.609749629 +0000 UTC m=+4067.607438219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-maas-api-tls") pod "maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77" (UID: "71c22cf4-9ac4-4ad9-a59e-27afb7bee488") : secret "maas-api-e2e-inf-a-1dd5d5-serving-cert" not found
Nov 02 20:56:26.110204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.110173    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:26.112631 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.112603    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-metrics-certs\") pod \"maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:26.169742 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.169711    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7nd4\" (UniqueName: \"kubernetes.io/projected/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-kube-api-access-j7nd4\") pod \"maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:26.276675 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.276641    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:26.452240 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.452156    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:26.629061 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.629015    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:26.632167 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.632133    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:26.773602 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:26.773564    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:27.143125 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:56:27.143093    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c22cf4_9ac4_4ad9_a59e_27afb7bee488.slice/crio-c0c9393409ca4706247dcd319bd5b56e72040605d229a6fdd099c26f3f2cba2b WatchSource:0}: Error finding container c0c9393409ca4706247dcd319bd5b56e72040605d229a6fdd099c26f3f2cba2b: Status 404 returned error can't find the container with id c0c9393409ca4706247dcd319bd5b56e72040605d229a6fdd099c26f3f2cba2b
Nov 02 20:56:27.145637 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:27.145547    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"]
Nov 02 20:56:27.359860 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:56:27.359825    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:56:27.884390 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:27.884354    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77" event={"ID":"71c22cf4-9ac4-4ad9-a59e-27afb7bee488","Type":"ContainerStarted","Data":"c0c9393409ca4706247dcd319bd5b56e72040605d229a6fdd099c26f3f2cba2b"}
Nov 02 20:56:28.892742 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:28.892704    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77" event={"ID":"71c22cf4-9ac4-4ad9-a59e-27afb7bee488","Type":"ContainerStarted","Data":"8d2e6a19603b5d537e0c383453e4098073e27fece59068237381d42ddaa778ce"}
Nov 02 20:56:28.893318 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:28.892950    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:29.009957 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:29.009895    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77" podStartSLOduration=3.172610489 podStartE2EDuration="4.009881208s" totalImagesPullingTime="837.270719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:56:29.009017957 +0000 UTC m=+4070.006706577" watchObservedRunningTime="2026-11-02 20:56:29.009881208 +0000 UTC m=+4070.007569834"
Nov 02 20:56:31.315481 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:31.315445    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:34.904322 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:34.904292    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:56:36.269540 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:36.269500    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:41.166856 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:41.166818    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:46.000051 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:46.000008    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:50.902049 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:50.902014    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:56.067138 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:56:56.067096    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:56:57.359864 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:56:57.359828    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:57:00.936372 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:00.936324    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:05.975560 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:05.975499    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:07.716591 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:07.716553    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:08.466704 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:08.466665    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:10.953659 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:10.953616    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:12.581991 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:12.581956    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:14.409712 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:14.409675    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:57:15.959360 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:15.959320    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:20.010037 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:20.010003    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:20.459548 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:20.459501    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:21.040080 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:21.040045    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:21.565212 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:21.565165    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:25.267063 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:25.267013    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:25.452357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:25.452319    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:25.683677 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:25.683598    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:26.204279 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:26.204243    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:27.358420 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:57:27.358388    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:57:27.701601 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:27.701507    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:28.108058 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.108011    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:28.563285 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.563245    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"]
Nov 02 20:57:28.567454 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.567429    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.629792 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.622790    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-801923-openshift-default-dockercfg-pr7s2\""
Nov 02 20:57:28.677586 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.677546    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"]
Nov 02 20:57:28.725204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.725169    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-certs\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.725410 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.725240    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-credential-socket\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.725410 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.725316    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-podinfo\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.725410 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.725357    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-socket\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.725410 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.725374    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rll\" (UniqueName: \"kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-kube-api-access-59rll\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.725660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.725459    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-token\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.725660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.725493    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-data\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.725660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.725527    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istiod-ca-cert\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.725660 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.725623    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-envoy\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.772558 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.772517    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:28.826697 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.826610    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-token\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.826697 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.826663    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-data\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.826996 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.826714    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istiod-ca-cert\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.826996 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.826810    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-envoy\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.826996 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.826890    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-certs\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.826996 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.826979    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-credential-socket\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.827232 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.827010    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-podinfo\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.827232 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.827049    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-socket\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.827232 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.827084    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rll\" (UniqueName: \"kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-kube-api-access-59rll\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.827232 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.827137    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-data\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.827477 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.827449    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-credential-socket\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.827644 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.827624    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istiod-ca-cert\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.827728 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.827666    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-certs\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.827728 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.827696    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-socket\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.830278 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.830251    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-podinfo\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.833011 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.832987    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-envoy\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.887915 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.887883    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-token\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:28.888065 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:28.887986    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rll\" (UniqueName: \"kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-kube-api-access-59rll\") pod \"e2e-inf-b-801923-openshift-default-699785fb7-ts775\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") " pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:29.048210 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:29.048157    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:29.181215 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:29.181137    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:29.242816 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:29.238698    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:29.412029 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:29.411993    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:29.641985 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:29.641950    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"]
Nov 02 20:57:29.649403 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:29.649363    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:57:29.649672 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:29.649644    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:57:29.649881 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:29.649842    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:57:30.277639 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:30.277604    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775" event={"ID":"232a5dd5-ff8a-4aa4-960c-ccaf79722eda","Type":"ContainerStarted","Data":"bb94a61cdfb42855a61bc8eb50a2b45eb8f6f603cfe7810e81ee0b20431c6630"}
Nov 02 20:57:30.277857 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:30.277653    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775" event={"ID":"232a5dd5-ff8a-4aa4-960c-ccaf79722eda","Type":"ContainerStarted","Data":"bb7ead1f51928aaab29772bf3f9dbea33ad699d37db449ccbf91356e725bdf03"}
Nov 02 20:57:30.397623 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:30.397566    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775" podStartSLOduration=2.397550091 podStartE2EDuration="2.397550091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:57:30.39543955 +0000 UTC m=+4131.393128159" watchObservedRunningTime="2026-11-02 20:57:30.397550091 +0000 UTC m=+4131.395238700"
Nov 02 20:57:31.182314 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.182280    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:31.188390 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.188365    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:31.213427 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.213404    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"]
Nov 02 20:57:31.217252 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.217236    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:31.269793 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.269758    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"]
Nov 02 20:57:31.283683 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.283654    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:31.284523 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.284503    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 20:57:31.292562 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.292544    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4530c5-serving-cert\""
Nov 02 20:57:31.359986 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.359961    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-maas-api-tls\") pod \"maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:31.360127 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.360004    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-metrics-certs\") pod \"maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:31.360194 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.360173    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4hhw\" (UniqueName: \"kubernetes.io/projected/dffa4399-a7ef-4f19-8177-42c954bfbc40-kube-api-access-c4hhw\") pod \"maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:31.360245 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.360233    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffa4399-a7ef-4f19-8177-42c954bfbc40-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:31.461874 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.461792    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4hhw\" (UniqueName: \"kubernetes.io/projected/dffa4399-a7ef-4f19-8177-42c954bfbc40-kube-api-access-c4hhw\") pod \"maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:31.462061 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.461962    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffa4399-a7ef-4f19-8177-42c954bfbc40-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:31.462061 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.461999    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-maas-api-tls\") pod \"maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:31.462061 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.462039    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-metrics-certs\") pod \"maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:31.462271 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:57:31.462118    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-serving-cert: secret "maas-api-e2e-inf-a-4530c5-serving-cert" not found
Nov 02 20:57:31.462271 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:57:31.462187    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-maas-api-tls podName:dffa4399-a7ef-4f19-8177-42c954bfbc40 nodeName:}" failed. No retries permitted until 2026-11-02 20:57:31.96216504 +0000 UTC m=+4132.959853646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-maas-api-tls") pod "maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn" (UID: "dffa4399-a7ef-4f19-8177-42c954bfbc40") : secret "maas-api-e2e-inf-a-4530c5-serving-cert" not found
Nov 02 20:57:31.462657 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.462634    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffa4399-a7ef-4f19-8177-42c954bfbc40-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:31.464451 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.464431    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-metrics-certs\") pod \"maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:31.492589 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.492562    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:31.521353 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.521319    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4hhw\" (UniqueName: \"kubernetes.io/projected/dffa4399-a7ef-4f19-8177-42c954bfbc40-kube-api-access-c4hhw\") pod \"maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:31.668020 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.667987    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:31.952631 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.952589    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:31.967758 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.967733    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-maas-api-tls\") pod \"maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:31.970100 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:31.970075    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-maas-api-tls\") pod \"maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:32.131264 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:32.131229    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:32.498035 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:57:32.498008    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffa4399_a7ef_4f19_8177_42c954bfbc40.slice/crio-8f87ce01c05dc4059a57aae1c73994565a7b6db4094051e248658ec6060b0cba WatchSource:0}: Error finding container 8f87ce01c05dc4059a57aae1c73994565a7b6db4094051e248658ec6060b0cba: Status 404 returned error can't find the container with id 8f87ce01c05dc4059a57aae1c73994565a7b6db4094051e248658ec6060b0cba
Nov 02 20:57:32.501446 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:32.501423    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"]
Nov 02 20:57:33.296949 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:33.296915    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn" event={"ID":"dffa4399-a7ef-4f19-8177-42c954bfbc40","Type":"ContainerStarted","Data":"8f87ce01c05dc4059a57aae1c73994565a7b6db4094051e248658ec6060b0cba"}
Nov 02 20:57:33.846283 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:33.846244    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:34.071331 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:34.071291    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:34.304654 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:34.304614    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn" event={"ID":"dffa4399-a7ef-4f19-8177-42c954bfbc40","Type":"ContainerStarted","Data":"23f5de4e5362a3d161a0cd5fcebe84c0a4b750ae25104befa3beec6a4ccd9229"}
Nov 02 20:57:34.304866 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:34.304710    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:34.424875 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:34.424823    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn" podStartSLOduration=2.494415079 podStartE2EDuration="3.424808664s" totalImagesPullingTime="930.393585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:57:34.422355741 +0000 UTC m=+4135.420044351" watchObservedRunningTime="2026-11-02 20:57:34.424808664 +0000 UTC m=+4135.422497273"
Nov 02 20:57:36.871655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:36.871621    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:40.268184 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:40.268140    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:40.315644 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:40.315616    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 20:57:40.446315 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:40.446281    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:41.863615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:41.863578    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:46.745986 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:46.745930    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:51.598576 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:51.598490    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:56.676790 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:57:56.676728    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:57:57.357851 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:57:57.357816    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:58:01.293812 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:01.293757    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:04.383588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:04.383550    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:06.110211 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:06.110176    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:58:06.575103 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:06.575062    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:08.035934 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:08.035901    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:08.793677 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:08.793641    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:10.248535 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:10.248494    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:10.421458 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:10.421412    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:11.862684 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:11.862646    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:15.797979 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:15.797948    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:18.223204 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:18.223165    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:21.735560 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:21.735514    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:26.664539 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:26.664490    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:27.358804 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:58:27.358750    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:58:31.558906 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:31.558868    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:36.382345 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:36.382298    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:41.274908 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:41.274864    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:46.177562 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:46.177520    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:50.964382 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:50.964323    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:55.476811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.476780    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"]
Nov 02 20:58:55.480698 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.480677    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:55.533424 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.533386    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"]
Nov 02 20:58:55.581744 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.581712    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e19d40-serving-cert\""
Nov 02 20:58:55.611588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.611507    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-metrics-certs\") pod \"maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:55.611804 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.611629    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jtwk\" (UniqueName: \"kubernetes.io/projected/5840d9e3-7b79-408a-ab57-2dd8ca622771-kube-api-access-4jtwk\") pod \"maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:55.611894 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.611835    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5840d9e3-7b79-408a-ab57-2dd8ca622771-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:55.612032 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.612006    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-maas-api-tls\") pod \"maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:55.688398 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.688355    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:55.713395 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.713353    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-maas-api-tls\") pod \"maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:55.713395 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.713399    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-metrics-certs\") pod \"maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:55.713658 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.713458    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jtwk\" (UniqueName: \"kubernetes.io/projected/5840d9e3-7b79-408a-ab57-2dd8ca622771-kube-api-access-4jtwk\") pod \"maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:55.713658 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:58:55.713533    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-serving-cert: secret "maas-api-e2e-inf-b-e19d40-serving-cert" not found
Nov 02 20:58:55.713658 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.713584    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5840d9e3-7b79-408a-ab57-2dd8ca622771-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:55.713658 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:58:55.713618    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-maas-api-tls podName:5840d9e3-7b79-408a-ab57-2dd8ca622771 nodeName:}" failed. No retries permitted until 2026-11-02 20:58:56.213596504 +0000 UTC m=+4217.211285108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-maas-api-tls") pod "maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc" (UID: "5840d9e3-7b79-408a-ab57-2dd8ca622771") : secret "maas-api-e2e-inf-b-e19d40-serving-cert" not found
Nov 02 20:58:55.714385 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.714358    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5840d9e3-7b79-408a-ab57-2dd8ca622771-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:55.716609 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.716587    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-metrics-certs\") pod \"maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:55.772287 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.772258    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jtwk\" (UniqueName: \"kubernetes.io/projected/5840d9e3-7b79-408a-ab57-2dd8ca622771-kube-api-access-4jtwk\") pod \"maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:55.912717 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:55.912679    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:56.167658 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:56.167568    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:58:56.220655 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:56.220622    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-maas-api-tls\") pod \"maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:56.223027 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:56.223006    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-maas-api-tls\") pod \"maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:56.394957 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:56.394928    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:56.763567 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:58:56.763535    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5840d9e3_7b79_408a_ab57_2dd8ca622771.slice/crio-956932525e61fbfe80d30b34a86173a85316c9f1a463e9862659933331708f6b WatchSource:0}: Error finding container 956932525e61fbfe80d30b34a86173a85316c9f1a463e9862659933331708f6b: Status 404 returned error can't find the container with id 956932525e61fbfe80d30b34a86173a85316c9f1a463e9862659933331708f6b
Nov 02 20:58:56.766726 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:56.766702    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"]
Nov 02 20:58:56.790795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:56.788412    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc" event={"ID":"5840d9e3-7b79-408a-ab57-2dd8ca622771","Type":"ContainerStarted","Data":"956932525e61fbfe80d30b34a86173a85316c9f1a463e9862659933331708f6b"}
Nov 02 20:58:57.359005 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:58:57.358968    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:58:57.798507 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:57.798465    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc" event={"ID":"5840d9e3-7b79-408a-ab57-2dd8ca622771","Type":"ContainerStarted","Data":"332efa168b25a1c9ce1aa60177b2df5195958607eeeae830f4c5568db4185866"}
Nov 02 20:58:57.799047 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:57.798548    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:58:57.916638 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:58:57.916540    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc" podStartSLOduration=2.070150806 podStartE2EDuration="2.916525325s" totalImagesPullingTime="846.374519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:58:57.913502852 +0000 UTC m=+4218.911191461" watchObservedRunningTime="2026-11-02 20:58:57.916525325 +0000 UTC m=+4218.914213933"
Nov 02 20:59:00.860063 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:00.860030    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:03.810659 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:03.810624    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 20:59:05.359522 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.359490    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"]
Nov 02 20:59:05.363721 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.363704    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:05.416946 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.416914    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"]
Nov 02 20:59:05.464517 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.464481    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-801923-serving-cert\""
Nov 02 20:59:05.527031 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.526999    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvnn\" (UniqueName: \"kubernetes.io/projected/13c4fdb1-f36b-4674-91b6-e84a17df72ce-kube-api-access-wlvnn\") pod \"maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:05.527233 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.527058    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c4fdb1-f36b-4674-91b6-e84a17df72ce-openshift-service-ca\") pod \"maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:05.527233 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.527172    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-maas-api-tls\") pod \"maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:05.527233 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.527208    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-metrics-certs\") pod \"maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:05.628413 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.628329    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-maas-api-tls\") pod \"maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:05.628413 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.628371    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-metrics-certs\") pod \"maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:05.628620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.628485    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvnn\" (UniqueName: \"kubernetes.io/projected/13c4fdb1-f36b-4674-91b6-e84a17df72ce-kube-api-access-wlvnn\") pod \"maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:05.628620 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.628523    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c4fdb1-f36b-4674-91b6-e84a17df72ce-openshift-service-ca\") pod \"maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:05.629166 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.629142    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c4fdb1-f36b-4674-91b6-e84a17df72ce-openshift-service-ca\") pod \"maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:05.630907 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.630879    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-maas-api-tls\") pod \"maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:05.630907 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.630893    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-metrics-certs\") pod \"maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:05.656214 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.656186    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:05.685859 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.685832    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvnn\" (UniqueName: \"kubernetes.io/projected/13c4fdb1-f36b-4674-91b6-e84a17df72ce-kube-api-access-wlvnn\") pod \"maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:05.832187 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.832152    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:05.976830 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:05.976736    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:06.353451 ip-10-0-134-196 kubenswrapper[5224]: W1102 20:59:06.353413    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c4fdb1_f36b_4674_91b6_e84a17df72ce.slice/crio-94f23b939ccb7015bc444b1519b09c05a37ec4dc56bd1f63c5e0d2fe5152f82c WatchSource:0}: Error finding container 94f23b939ccb7015bc444b1519b09c05a37ec4dc56bd1f63c5e0d2fe5152f82c: Status 404 returned error can't find the container with id 94f23b939ccb7015bc444b1519b09c05a37ec4dc56bd1f63c5e0d2fe5152f82c
Nov 02 20:59:06.356811 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:06.356783    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"]
Nov 02 20:59:06.863092 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:06.863055    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw" event={"ID":"13c4fdb1-f36b-4674-91b6-e84a17df72ce","Type":"ContainerStarted","Data":"94f23b939ccb7015bc444b1519b09c05a37ec4dc56bd1f63c5e0d2fe5152f82c"}
Nov 02 20:59:07.874961 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:07.874929    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw" event={"ID":"13c4fdb1-f36b-4674-91b6-e84a17df72ce","Type":"ContainerStarted","Data":"e6534369605532efe0d64c46f51d2899282fd77ee5e4dd0bdc226cb4e706540d"}
Nov 02 20:59:07.875357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:07.875041    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:07.992876 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:07.992829    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw" podStartSLOduration=2.14813924 podStartE2EDuration="2.992815949s" totalImagesPullingTime="844.676709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:59:07.992198585 +0000 UTC m=+4228.989887178" watchObservedRunningTime="2026-11-02 20:59:07.992815949 +0000 UTC m=+4228.990504557"
Nov 02 20:59:08.388309 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:08.388276    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:09.394088 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:09.394053    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:09.712078 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:09.711987    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:10.664906 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:10.664874    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:12.948020 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:12.947984    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 20:59:13.887262 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:13.887229    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 20:59:15.559889 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:15.559850    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:15.779977 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:15.779941    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:16.082403 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:16.082370    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"]
Nov 02 20:59:16.190510 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:16.190479    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"]
Nov 02 20:59:16.570693 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:16.570658    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:16.747924 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:16.747889    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:20.773389 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:20.773296    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:20.988787 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:20.988726    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:21.082951 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.082870    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm" podUID="045ccbb5-32a4-4ccb-8ed7-f17fff958929" containerName="payload-processing" containerID="cri-o://08ed3530bac3b1e0fb177ba8e3f4ca6bed2e4c5ef75d801a303f137f3feb1a6b" gracePeriod=25
Nov 02 20:59:21.191129 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.191097    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh" podUID="0a0b0bcc-fa85-4dad-b946-2d9917a9dce6" containerName="payload-pre-processing" containerID="cri-o://8c16cfb5a40dca851c05fb58e2962aef02db9fb4eb81638bbc8be91e6a1d6d8a" gracePeriod=25
Nov 02 20:59:21.357534 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.357505    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"
Nov 02 20:59:21.409948 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.409840    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045ccbb5-32a4-4ccb-8ed7-f17fff958929-kube-api-access-vtvn4\" (UniqueName: \"kubernetes.io/projected/045ccbb5-32a4-4ccb-8ed7-f17fff958929-kube-api-access-vtvn4\") pod \"045ccbb5-32a4-4ccb-8ed7-f17fff958929\" (UID: \"045ccbb5-32a4-4ccb-8ed7-f17fff958929\") "
Nov 02 20:59:21.410125 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.410036    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045ccbb5-32a4-4ccb-8ed7-f17fff958929-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045ccbb5-32a4-4ccb-8ed7-f17fff958929-plugins-config-volume\") pod \"045ccbb5-32a4-4ccb-8ed7-f17fff958929\" (UID: \"045ccbb5-32a4-4ccb-8ed7-f17fff958929\") "
Nov 02 20:59:21.410509 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.410476    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045ccbb5-32a4-4ccb-8ed7-f17fff958929-plugins-config-volume" pod "045ccbb5-32a4-4ccb-8ed7-f17fff958929" (UID: "045ccbb5-32a4-4ccb-8ed7-f17fff958929"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:59:21.412119 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.412086    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045ccbb5-32a4-4ccb-8ed7-f17fff958929-kube-api-access-vtvn4" pod "045ccbb5-32a4-4ccb-8ed7-f17fff958929" (UID: "045ccbb5-32a4-4ccb-8ed7-f17fff958929"). InnerVolumeSpecName "kube-api-access-vtvn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:59:21.458186 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.458161    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"
Nov 02 20:59:21.511588 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.511558    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-plugins-config-volume\") pod \"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6\" (UID: \"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6\") "
Nov 02 20:59:21.511737 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.511663    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-kube-api-access-tlb7n\" (UniqueName: \"kubernetes.io/projected/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-kube-api-access-tlb7n\") pod \"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6\" (UID: \"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6\") "
Nov 02 20:59:21.511931 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.511909    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-plugins-config-volume" pod "0a0b0bcc-fa85-4dad-b946-2d9917a9dce6" (UID: "0a0b0bcc-fa85-4dad-b946-2d9917a9dce6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:59:21.512121 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.512102    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045ccbb5-32a4-4ccb-8ed7-f17fff958929-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:21.512121 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.512118    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtvn4\" (UniqueName: \"kubernetes.io/projected/045ccbb5-32a4-4ccb-8ed7-f17fff958929-kube-api-access-vtvn4\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:21.512267 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.512128    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:21.513708 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.513682    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-kube-api-access-tlb7n" pod "0a0b0bcc-fa85-4dad-b946-2d9917a9dce6" (UID: "0a0b0bcc-fa85-4dad-b946-2d9917a9dce6"). InnerVolumeSpecName "kube-api-access-tlb7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:59:21.613663 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.613581    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlb7n\" (UniqueName: \"kubernetes.io/projected/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6-kube-api-access-tlb7n\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:21.961182 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.961091    5224 generic.go:356] "Generic (PLEG): container finished" podID="0a0b0bcc-fa85-4dad-b946-2d9917a9dce6" containerID="8c16cfb5a40dca851c05fb58e2962aef02db9fb4eb81638bbc8be91e6a1d6d8a" exitCode=0
Nov 02 20:59:21.961182 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.961144    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"
Nov 02 20:59:21.961182 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.961172    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh" event={"ID":"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6","Type":"ContainerDied","Data":"8c16cfb5a40dca851c05fb58e2962aef02db9fb4eb81638bbc8be91e6a1d6d8a"}
Nov 02 20:59:21.961795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.961273    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh" event={"ID":"0a0b0bcc-fa85-4dad-b946-2d9917a9dce6","Type":"ContainerDied","Data":"f94a9eba03b981f687b0f31cbda6c449fa6f0c8a64b4baf31a6c69b306beae1a"}
Nov 02 20:59:21.961795 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.961311    5224 scope.go:122] "RemoveContainer" containerID="8c16cfb5a40dca851c05fb58e2962aef02db9fb4eb81638bbc8be91e6a1d6d8a"
Nov 02 20:59:21.962615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.962589    5224 generic.go:356] "Generic (PLEG): container finished" podID="045ccbb5-32a4-4ccb-8ed7-f17fff958929" containerID="08ed3530bac3b1e0fb177ba8e3f4ca6bed2e4c5ef75d801a303f137f3feb1a6b" exitCode=0
Nov 02 20:59:21.962734 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.962640    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"
Nov 02 20:59:21.962734 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.962679    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm" event={"ID":"045ccbb5-32a4-4ccb-8ed7-f17fff958929","Type":"ContainerDied","Data":"08ed3530bac3b1e0fb177ba8e3f4ca6bed2e4c5ef75d801a303f137f3feb1a6b"}
Nov 02 20:59:21.962734 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.962715    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm" event={"ID":"045ccbb5-32a4-4ccb-8ed7-f17fff958929","Type":"ContainerDied","Data":"d9b031af4e16315e065ea56254f47756138a4e2891bfdc1ebdb5bf2944996569"}
Nov 02 20:59:21.973039 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.973019    5224 scope.go:122] "RemoveContainer" containerID="8c16cfb5a40dca851c05fb58e2962aef02db9fb4eb81638bbc8be91e6a1d6d8a"
Nov 02 20:59:21.973310 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:59:21.973292    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c16cfb5a40dca851c05fb58e2962aef02db9fb4eb81638bbc8be91e6a1d6d8a\": container with ID starting with 8c16cfb5a40dca851c05fb58e2962aef02db9fb4eb81638bbc8be91e6a1d6d8a not found: ID does not exist" containerID="8c16cfb5a40dca851c05fb58e2962aef02db9fb4eb81638bbc8be91e6a1d6d8a"
Nov 02 20:59:21.973393 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.973322    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c16cfb5a40dca851c05fb58e2962aef02db9fb4eb81638bbc8be91e6a1d6d8a"} err="failed to get container status \"8c16cfb5a40dca851c05fb58e2962aef02db9fb4eb81638bbc8be91e6a1d6d8a\": rpc error: code = NotFound desc = could not find container \"8c16cfb5a40dca851c05fb58e2962aef02db9fb4eb81638bbc8be91e6a1d6d8a\": container with ID starting with 8c16cfb5a40dca851c05fb58e2962aef02db9fb4eb81638bbc8be91e6a1d6d8a not found: ID does not exist"
Nov 02 20:59:21.973393 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.973340    5224 scope.go:122] "RemoveContainer" containerID="08ed3530bac3b1e0fb177ba8e3f4ca6bed2e4c5ef75d801a303f137f3feb1a6b"
Nov 02 20:59:21.981747 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.981727    5224 scope.go:122] "RemoveContainer" containerID="08ed3530bac3b1e0fb177ba8e3f4ca6bed2e4c5ef75d801a303f137f3feb1a6b"
Nov 02 20:59:21.982027 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:59:21.982006    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ed3530bac3b1e0fb177ba8e3f4ca6bed2e4c5ef75d801a303f137f3feb1a6b\": container with ID starting with 08ed3530bac3b1e0fb177ba8e3f4ca6bed2e4c5ef75d801a303f137f3feb1a6b not found: ID does not exist" containerID="08ed3530bac3b1e0fb177ba8e3f4ca6bed2e4c5ef75d801a303f137f3feb1a6b"
Nov 02 20:59:21.982083 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:21.982032    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ed3530bac3b1e0fb177ba8e3f4ca6bed2e4c5ef75d801a303f137f3feb1a6b"} err="failed to get container status \"08ed3530bac3b1e0fb177ba8e3f4ca6bed2e4c5ef75d801a303f137f3feb1a6b\": rpc error: code = NotFound desc = could not find container \"08ed3530bac3b1e0fb177ba8e3f4ca6bed2e4c5ef75d801a303f137f3feb1a6b\": container with ID starting with 08ed3530bac3b1e0fb177ba8e3f4ca6bed2e4c5ef75d801a303f137f3feb1a6b not found: ID does not exist"
Nov 02 20:59:22.129021 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:22.128992    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"]
Nov 02 20:59:22.169366 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:22.169341    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4530c5-f48bf8489-txpmm"]
Nov 02 20:59:22.281884 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:22.281857    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"]
Nov 02 20:59:22.285712 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:22.285691    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4530c5-59596dc99-cgdjh"]
Nov 02 20:59:23.167687 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:23.167655    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045ccbb5-32a4-4ccb-8ed7-f17fff958929" path="/var/lib/kubelet/pods/045ccbb5-32a4-4ccb-8ed7-f17fff958929/volumes"
Nov 02 20:59:23.168117 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:23.168100    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0b0bcc-fa85-4dad-b946-2d9917a9dce6" path="/var/lib/kubelet/pods/0a0b0bcc-fa85-4dad-b946-2d9917a9dce6/volumes"
Nov 02 20:59:25.657180 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:25.657144    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:27.358540 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:59:27.358502    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:59:30.542564 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:30.542530    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:34.606892 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:34.606862    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"]
Nov 02 20:59:34.607462 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:34.607059    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77" podUID="71c22cf4-9ac4-4ad9-a59e-27afb7bee488" containerName="maas-api" containerID="cri-o://8d2e6a19603b5d537e0c383453e4098073e27fece59068237381d42ddaa778ce" gracePeriod=30
Nov 02 20:59:34.752418 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:34.752381    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:34.895476 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:34.895446    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:59:34.959900 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:34.959867    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-kube-api-access-j7nd4\" (UniqueName: \"kubernetes.io/projected/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-kube-api-access-j7nd4\") pod \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") "
Nov 02 20:59:34.960091 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:34.960001    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-maas-api-tls\") pod \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") "
Nov 02 20:59:34.960091 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:34.960055    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-openshift-service-ca\") pod \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") "
Nov 02 20:59:34.960205 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:34.960136    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-metrics-certs\") pod \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\" (UID: \"71c22cf4-9ac4-4ad9-a59e-27afb7bee488\") "
Nov 02 20:59:34.960694 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:34.960631    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-openshift-service-ca" pod "71c22cf4-9ac4-4ad9-a59e-27afb7bee488" (UID: "71c22cf4-9ac4-4ad9-a59e-27afb7bee488"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:59:34.962322 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:34.962297    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-maas-api-tls" pod "71c22cf4-9ac4-4ad9-a59e-27afb7bee488" (UID: "71c22cf4-9ac4-4ad9-a59e-27afb7bee488"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:59:34.962641 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:34.962612    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-kube-api-access-j7nd4" pod "71c22cf4-9ac4-4ad9-a59e-27afb7bee488" (UID: "71c22cf4-9ac4-4ad9-a59e-27afb7bee488"). InnerVolumeSpecName "kube-api-access-j7nd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:59:34.963527 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:34.963501    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-metrics-certs" pod "71c22cf4-9ac4-4ad9-a59e-27afb7bee488" (UID: "71c22cf4-9ac4-4ad9-a59e-27afb7bee488"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:59:35.049554 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:35.049516    5224 generic.go:356] "Generic (PLEG): container finished" podID="71c22cf4-9ac4-4ad9-a59e-27afb7bee488" containerID="8d2e6a19603b5d537e0c383453e4098073e27fece59068237381d42ddaa778ce" exitCode=0
Nov 02 20:59:35.049799 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:35.049579    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77" event={"ID":"71c22cf4-9ac4-4ad9-a59e-27afb7bee488","Type":"ContainerDied","Data":"8d2e6a19603b5d537e0c383453e4098073e27fece59068237381d42ddaa778ce"}
Nov 02 20:59:35.049799 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:35.049597    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"
Nov 02 20:59:35.049799 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:35.049633    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77" event={"ID":"71c22cf4-9ac4-4ad9-a59e-27afb7bee488","Type":"ContainerDied","Data":"c0c9393409ca4706247dcd319bd5b56e72040605d229a6fdd099c26f3f2cba2b"}
Nov 02 20:59:35.049799 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:35.049663    5224 scope.go:122] "RemoveContainer" containerID="8d2e6a19603b5d537e0c383453e4098073e27fece59068237381d42ddaa778ce"
Nov 02 20:59:35.062095 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:35.062006    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7nd4\" (UniqueName: \"kubernetes.io/projected/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-kube-api-access-j7nd4\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:35.062095 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:35.062041    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:35.062095 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:35.062057    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:35.062095 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:35.062071    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c22cf4-9ac4-4ad9-a59e-27afb7bee488-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:35.072386 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:35.072367    5224 scope.go:122] "RemoveContainer" containerID="8d2e6a19603b5d537e0c383453e4098073e27fece59068237381d42ddaa778ce"
Nov 02 20:59:35.072719 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:59:35.072699    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d2e6a19603b5d537e0c383453e4098073e27fece59068237381d42ddaa778ce\": container with ID starting with 8d2e6a19603b5d537e0c383453e4098073e27fece59068237381d42ddaa778ce not found: ID does not exist" containerID="8d2e6a19603b5d537e0c383453e4098073e27fece59068237381d42ddaa778ce"
Nov 02 20:59:35.072788 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:35.072728    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d2e6a19603b5d537e0c383453e4098073e27fece59068237381d42ddaa778ce"} err="failed to get container status \"8d2e6a19603b5d537e0c383453e4098073e27fece59068237381d42ddaa778ce\": rpc error: code = NotFound desc = could not find container \"8d2e6a19603b5d537e0c383453e4098073e27fece59068237381d42ddaa778ce\": container with ID starting with 8d2e6a19603b5d537e0c383453e4098073e27fece59068237381d42ddaa778ce not found: ID does not exist"
Nov 02 20:59:35.217535 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:35.217452    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"]
Nov 02 20:59:35.221241 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:35.221210    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dd5d5-646cfb8fc7-5jk77"]
Nov 02 20:59:36.600331 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:36.600288    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:36.919088 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:36.919002    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:37.168268 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:37.168234    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c22cf4-9ac4-4ad9-a59e-27afb7bee488" path="/var/lib/kubelet/pods/71c22cf4-9ac4-4ad9-a59e-27afb7bee488/volumes"
Nov 02 20:59:37.228741 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:37.228656    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:38.154968 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:38.154933    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:38.450909 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:38.450802    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:38.628599 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:38.628564    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:39.995808 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:39.995757    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:40.171112 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:40.171063    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:42.073733 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:42.073694    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:46.970878 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:46.970840    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:47.156615 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:47.156569    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:51.874789 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:51.871720    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:56.716159 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:56.716124    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 20:59:57.357484 ip-10-0-134-196 kubenswrapper[5224]: E1102 20:59:57.357450    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 20:59:58.212357 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:58.212325    5224 generic.go:356] "Generic (PLEG): container finished" podID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerID="bb94a61cdfb42855a61bc8eb50a2b45eb8f6f603cfe7810e81ee0b20431c6630" exitCode=0
Nov 02 20:59:58.212827 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:58.212401    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775" event={"ID":"232a5dd5-ff8a-4aa4-960c-ccaf79722eda","Type":"ContainerDied","Data":"bb94a61cdfb42855a61bc8eb50a2b45eb8f6f603cfe7810e81ee0b20431c6630"}
Nov 02 20:59:58.212893 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:58.212875    5224 scope.go:122] "RemoveContainer" containerID="bb94a61cdfb42855a61bc8eb50a2b45eb8f6f603cfe7810e81ee0b20431c6630"
Nov 02 20:59:58.214613 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:58.214580    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:59:58.221225 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:58.215160    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:59:58.221225 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:58.215338    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 20:59:59.220418 ip-10-0-134-196 kubenswrapper[5224]: I1102 20:59:59.220385    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775" event={"ID":"232a5dd5-ff8a-4aa4-960c-ccaf79722eda","Type":"ContainerStarted","Data":"0a1a5eac6f67b6c7ad4594847bee89e0bdbb45ccd3109bb46ab7150f77d9c690"}
Nov 02 21:00:00.182289 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.182248    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 21:00:00.183456 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.183430    5224 patch_prober.go:29] interesting pod/e2e-inf-b-801923-openshift-default-699785fb7-ts775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 21:00:00.183594 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.183476    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775" podUID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 21:00:00.191284 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.191258    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"]
Nov 02 21:00:00.191807 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.191791    5224 cpu_manager.go:414] "containerMap: removing container" podUID="045ccbb5-32a4-4ccb-8ed7-f17fff958929" containerName="payload-processing"
Nov 02 21:00:00.191901 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.191809    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045ccbb5-32a4-4ccb-8ed7-f17fff958929" containerName="payload-processing"
Nov 02 21:00:00.191901 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.191825    5224 cpu_manager.go:414] "containerMap: removing container" podUID="71c22cf4-9ac4-4ad9-a59e-27afb7bee488" containerName="maas-api"
Nov 02 21:00:00.191901 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.191832    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c22cf4-9ac4-4ad9-a59e-27afb7bee488" containerName="maas-api"
Nov 02 21:00:00.191901 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.191872    5224 cpu_manager.go:414] "containerMap: removing container" podUID="0a0b0bcc-fa85-4dad-b946-2d9917a9dce6" containerName="payload-pre-processing"
Nov 02 21:00:00.191901 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.191880    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0b0bcc-fa85-4dad-b946-2d9917a9dce6" containerName="payload-pre-processing"
Nov 02 21:00:00.192166 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.192046    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0b0bcc-fa85-4dad-b946-2d9917a9dce6" containerName="payload-pre-processing"
Nov 02 21:00:00.192166 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.192066    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="045ccbb5-32a4-4ccb-8ed7-f17fff958929" containerName="payload-processing"
Nov 02 21:00:00.192166 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.192077    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c22cf4-9ac4-4ad9-a59e-27afb7bee488" containerName="maas-api"
Nov 02 21:00:00.195492 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.195470    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"
Nov 02 21:00:00.210750 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.210726    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"]
Nov 02 21:00:00.214500 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.214484    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"]
Nov 02 21:00:00.214612 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.214596    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"
Nov 02 21:00:00.217908 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.217893    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"
Nov 02 21:00:00.242955 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.242927    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"]
Nov 02 21:00:00.246450 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.246430    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"]
Nov 02 21:00:00.246640 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.246623    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"
Nov 02 21:00:00.263060 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.262932    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"
Nov 02 21:00:00.291842 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.291810    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d66f8a-6db0-4cab-9c46-d28df83c9953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk\" (UID: \"89d66f8a-6db0-4cab-9c46-d28df83c9953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"
Nov 02 21:00:00.291980 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.291881    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xl5c\" (UniqueName: \"kubernetes.io/projected/89d66f8a-6db0-4cab-9c46-d28df83c9953-kube-api-access-5xl5c\") pod \"maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk\" (UID: \"89d66f8a-6db0-4cab-9c46-d28df83c9953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"
Nov 02 21:00:00.309230 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.309197    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"]
Nov 02 21:00:00.324022 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.323987    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 21:00:00.384289 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.384251    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"]
Nov 02 21:00:00.387594 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.387567    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"]
Nov 02 21:00:00.391515 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.391485    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"]
Nov 02 21:00:00.392682 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.392654    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d66f8a-6db0-4cab-9c46-d28df83c9953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk\" (UID: \"89d66f8a-6db0-4cab-9c46-d28df83c9953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"
Nov 02 21:00:00.392866 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.392700    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cd7435-3e8f-4e42-8c96-08f4eafa8437-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb\" (UID: \"72cd7435-3e8f-4e42-8c96-08f4eafa8437\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"
Nov 02 21:00:00.392866 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.392821    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pkxl\" (UniqueName: \"kubernetes.io/projected/72cd7435-3e8f-4e42-8c96-08f4eafa8437-kube-api-access-8pkxl\") pod \"maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb\" (UID: \"72cd7435-3e8f-4e42-8c96-08f4eafa8437\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"
Nov 02 21:00:00.393051 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.392873    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xl5c\" (UniqueName: \"kubernetes.io/projected/89d66f8a-6db0-4cab-9c46-d28df83c9953-kube-api-access-5xl5c\") pod \"maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk\" (UID: \"89d66f8a-6db0-4cab-9c46-d28df83c9953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"
Nov 02 21:00:00.393051 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.392950    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lf9r\" (UniqueName: \"kubernetes.io/projected/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-kube-api-access-9lf9r\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5\" (UID: \"5a4383ae-73a5-45a1-979d-16a9dbb7e83a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"
Nov 02 21:00:00.393051 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.393002    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5\" (UID: \"5a4383ae-73a5-45a1-979d-16a9dbb7e83a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"
Nov 02 21:00:00.393223 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.393084    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894220-rgdhr\" (UID: \"b6d2d907-945a-4fd8-bb27-5a4c78b2b122\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"
Nov 02 21:00:00.393223 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.393117    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv\" (UID: \"3c3ebb63-cca2-47c5-be89-1dbfdd34c581\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"
Nov 02 21:00:00.393223 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.393149    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frd9b\" (UniqueName: \"kubernetes.io/projected/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-kube-api-access-frd9b\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv\" (UID: \"3c3ebb63-cca2-47c5-be89-1dbfdd34c581\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"
Nov 02 21:00:00.393223 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.393220    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4gnv\" (UniqueName: \"kubernetes.io/projected/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-kube-api-access-m4gnv\") pod \"maas-api-key-cleanup-29894220-rgdhr\" (UID: \"b6d2d907-945a-4fd8-bb27-5a4c78b2b122\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"
Nov 02 21:00:00.393579 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.393554    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d66f8a-6db0-4cab-9c46-d28df83c9953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk\" (UID: \"89d66f8a-6db0-4cab-9c46-d28df83c9953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"
Nov 02 21:00:00.399218 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.399199    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"]
Nov 02 21:00:00.476212 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.476140    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xl5c\" (UniqueName: \"kubernetes.io/projected/89d66f8a-6db0-4cab-9c46-d28df83c9953-kube-api-access-5xl5c\") pod \"maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk\" (UID: \"89d66f8a-6db0-4cab-9c46-d28df83c9953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"
Nov 02 21:00:00.493927 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.493894    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894220-rgdhr\" (UID: \"b6d2d907-945a-4fd8-bb27-5a4c78b2b122\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"
Nov 02 21:00:00.494066 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.493933    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv\" (UID: \"3c3ebb63-cca2-47c5-be89-1dbfdd34c581\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"
Nov 02 21:00:00.494066 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.493955    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frd9b\" (UniqueName: \"kubernetes.io/projected/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-kube-api-access-frd9b\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv\" (UID: \"3c3ebb63-cca2-47c5-be89-1dbfdd34c581\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"
Nov 02 21:00:00.494066 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.494000    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gnv\" (UniqueName: \"kubernetes.io/projected/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-kube-api-access-m4gnv\") pod \"maas-api-key-cleanup-29894220-rgdhr\" (UID: \"b6d2d907-945a-4fd8-bb27-5a4c78b2b122\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"
Nov 02 21:00:00.494066 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.494060    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cd7435-3e8f-4e42-8c96-08f4eafa8437-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb\" (UID: \"72cd7435-3e8f-4e42-8c96-08f4eafa8437\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"
Nov 02 21:00:00.494277 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.494083    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pkxl\" (UniqueName: \"kubernetes.io/projected/72cd7435-3e8f-4e42-8c96-08f4eafa8437-kube-api-access-8pkxl\") pod \"maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb\" (UID: \"72cd7435-3e8f-4e42-8c96-08f4eafa8437\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"
Nov 02 21:00:00.494341 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.494284    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lf9r\" (UniqueName: \"kubernetes.io/projected/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-kube-api-access-9lf9r\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5\" (UID: \"5a4383ae-73a5-45a1-979d-16a9dbb7e83a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"
Nov 02 21:00:00.494402 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.494355    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5\" (UID: \"5a4383ae-73a5-45a1-979d-16a9dbb7e83a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"
Nov 02 21:00:00.494702 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.494680    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894220-rgdhr\" (UID: \"b6d2d907-945a-4fd8-bb27-5a4c78b2b122\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"
Nov 02 21:00:00.494833 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.494719    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv\" (UID: \"3c3ebb63-cca2-47c5-be89-1dbfdd34c581\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"
Nov 02 21:00:00.494833 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.494822    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cd7435-3e8f-4e42-8c96-08f4eafa8437-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb\" (UID: \"72cd7435-3e8f-4e42-8c96-08f4eafa8437\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"
Nov 02 21:00:00.495022 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.495003    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5\" (UID: \"5a4383ae-73a5-45a1-979d-16a9dbb7e83a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"
Nov 02 21:00:00.508553 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.508535    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"
Nov 02 21:00:00.578997 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.578961    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lf9r\" (UniqueName: \"kubernetes.io/projected/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-kube-api-access-9lf9r\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5\" (UID: \"5a4383ae-73a5-45a1-979d-16a9dbb7e83a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"
Nov 02 21:00:00.579298 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.579271    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4gnv\" (UniqueName: \"kubernetes.io/projected/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-kube-api-access-m4gnv\") pod \"maas-api-key-cleanup-29894220-rgdhr\" (UID: \"b6d2d907-945a-4fd8-bb27-5a4c78b2b122\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"
Nov 02 21:00:00.580171 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.580144    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frd9b\" (UniqueName: \"kubernetes.io/projected/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-kube-api-access-frd9b\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv\" (UID: \"3c3ebb63-cca2-47c5-be89-1dbfdd34c581\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"
Nov 02 21:00:00.580905 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.580885    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pkxl\" (UniqueName: \"kubernetes.io/projected/72cd7435-3e8f-4e42-8c96-08f4eafa8437-kube-api-access-8pkxl\") pod \"maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb\" (UID: \"72cd7435-3e8f-4e42-8c96-08f4eafa8437\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"
Nov 02 21:00:00.589102 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.585776    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"
Nov 02 21:00:00.594436 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.594386    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"
Nov 02 21:00:00.830830 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.830798    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"
Nov 02 21:00:00.834327 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.834302    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"
Nov 02 21:00:00.919336 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:00.912631    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"]
Nov 02 21:00:00.919336 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:00:00.914119    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d66f8a_6db0_4cab_9c46_d28df83c9953.slice/crio-a0e90505c2fe2f8c25a021652ae5d0b9dc06aac97468ec3caa10bb264db931cc WatchSource:0}: Error finding container a0e90505c2fe2f8c25a021652ae5d0b9dc06aac97468ec3caa10bb264db931cc: Status 404 returned error can't find the container with id a0e90505c2fe2f8c25a021652ae5d0b9dc06aac97468ec3caa10bb264db931cc
Nov 02 21:00:01.010195 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:01.010165    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"]
Nov 02 21:00:01.024115 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:00:01.024086    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cd7435_3e8f_4e42_8c96_08f4eafa8437.slice/crio-4c21da503e7d6a559b07da7e94a10cd673629251948ba4af1d41c3f75513eace WatchSource:0}: Error finding container 4c21da503e7d6a559b07da7e94a10cd673629251948ba4af1d41c3f75513eace: Status 404 returned error can't find the container with id 4c21da503e7d6a559b07da7e94a10cd673629251948ba4af1d41c3f75513eace
Nov 02 21:00:01.025548 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:01.025521    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"]
Nov 02 21:00:01.181716 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:01.181689    5224 patch_prober.go:29] interesting pod/e2e-inf-b-801923-openshift-default-699785fb7-ts775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 21:00:01.181949 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:01.181924    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775" podUID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 21:00:01.228047 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:00:01.228017    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4383ae_73a5_45a1_979d_16a9dbb7e83a.slice/crio-457d3224ca6af5a3af154fdfd5dd7fdb97b4ae735f2841675cde4aa2a79b58e0 WatchSource:0}: Error finding container 457d3224ca6af5a3af154fdfd5dd7fdb97b4ae735f2841675cde4aa2a79b58e0: Status 404 returned error can't find the container with id 457d3224ca6af5a3af154fdfd5dd7fdb97b4ae735f2841675cde4aa2a79b58e0
Nov 02 21:00:01.233298 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:01.233268    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"]
Nov 02 21:00:01.239049 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:01.239024    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr" event={"ID":"b6d2d907-945a-4fd8-bb27-5a4c78b2b122","Type":"ContainerStarted","Data":"ac9e70f242b455a0581556c13b2752d0dad5557dc1770682d284240a994c66ca"}
Nov 02 21:00:01.240185 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:01.240162    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb" event={"ID":"72cd7435-3e8f-4e42-8c96-08f4eafa8437","Type":"ContainerStarted","Data":"4c21da503e7d6a559b07da7e94a10cd673629251948ba4af1d41c3f75513eace"}
Nov 02 21:00:01.241405 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:01.241382    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk" event={"ID":"89d66f8a-6db0-4cab-9c46-d28df83c9953","Type":"ContainerStarted","Data":"a0e90505c2fe2f8c25a021652ae5d0b9dc06aac97468ec3caa10bb264db931cc"}
Nov 02 21:00:01.243328 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:01.243305    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"]
Nov 02 21:00:01.244231 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:00:01.244213    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3ebb63_cca2_47c5_be89_1dbfdd34c581.slice/crio-e3fcda57c08585b113d60d4991b6860474a0c3f6248d53575a4b3249791e7766 WatchSource:0}: Error finding container e3fcda57c08585b113d60d4991b6860474a0c3f6248d53575a4b3249791e7766: Status 404 returned error can't find the container with id e3fcda57c08585b113d60d4991b6860474a0c3f6248d53575a4b3249791e7766
Nov 02 21:00:01.284460 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:01.284434    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 21:00:01.529656 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:01.529612    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:02.181859 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.181828    5224 patch_prober.go:29] interesting pod/e2e-inf-b-801923-openshift-default-699785fb7-ts775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 21:00:02.182017 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.181883    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775" podUID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 21:00:02.267582 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.265090    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr" event={"ID":"b6d2d907-945a-4fd8-bb27-5a4c78b2b122","Type":"ContainerStarted","Data":"28c8a6ab64330133af2cd5cee4f7a1bf95ffb26ef9901700356dc7eb2526ad9d"}
Nov 02 21:00:02.280109 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.279323    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb" event={"ID":"72cd7435-3e8f-4e42-8c96-08f4eafa8437","Type":"ContainerStarted","Data":"108acd7b7d2adb44d31071c077576b4da8556a3aa28d68d3eb3ccfd2bd00cf97"}
Nov 02 21:00:02.281856 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.281828    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv" event={"ID":"3c3ebb63-cca2-47c5-be89-1dbfdd34c581","Type":"ContainerStarted","Data":"d0d4bf96ca92add5b03327a38fed8e66028d4162b9dc4a94efc45dfdaf5ffc42"}
Nov 02 21:00:02.282004 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.281873    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv" event={"ID":"3c3ebb63-cca2-47c5-be89-1dbfdd34c581","Type":"ContainerStarted","Data":"e3fcda57c08585b113d60d4991b6860474a0c3f6248d53575a4b3249791e7766"}
Nov 02 21:00:02.285789 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.283974    5224 generic.go:356] "Generic (PLEG): container finished" podID="89d66f8a-6db0-4cab-9c46-d28df83c9953" containerID="0a32904987b456da17b091207c927b0e2a32d8667b8222b39196ccb42a6a0ce9" exitCode=60
Nov 02 21:00:02.285789 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.284123    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk" event={"ID":"89d66f8a-6db0-4cab-9c46-d28df83c9953","Type":"ContainerDied","Data":"0a32904987b456da17b091207c927b0e2a32d8667b8222b39196ccb42a6a0ce9"}
Nov 02 21:00:02.285789 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.284553    5224 scope.go:122] "RemoveContainer" containerID="0a32904987b456da17b091207c927b0e2a32d8667b8222b39196ccb42a6a0ce9"
Nov 02 21:00:02.287200 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.287174    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5" event={"ID":"5a4383ae-73a5-45a1-979d-16a9dbb7e83a","Type":"ContainerStarted","Data":"fb26b9e4f31694118f09d4d273544d7cbddbe41ea0c12eb36000bae123c83c01"}
Nov 02 21:00:02.287289 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.287217    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5" event={"ID":"5a4383ae-73a5-45a1-979d-16a9dbb7e83a","Type":"ContainerStarted","Data":"457d3224ca6af5a3af154fdfd5dd7fdb97b4ae735f2841675cde4aa2a79b58e0"}
Nov 02 21:00:02.380740 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.380692    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr" podStartSLOduration=2.380677764 podStartE2EDuration="2.380677764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:02.378585189 +0000 UTC m=+4283.376273799" watchObservedRunningTime="2026-11-02 21:00:02.380677764 +0000 UTC m=+4283.378366373"
Nov 02 21:00:02.495478 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.495423    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5" podStartSLOduration=2.495409572 podStartE2EDuration="2.495409572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:02.494581183 +0000 UTC m=+4283.492269794" watchObservedRunningTime="2026-11-02 21:00:02.495409572 +0000 UTC m=+4283.493098226"
Nov 02 21:00:02.728364 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.728314    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb" podStartSLOduration=2.72829991 podStartE2EDuration="2.72829991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:02.726441679 +0000 UTC m=+4283.724130288" watchObservedRunningTime="2026-11-02 21:00:02.72829991 +0000 UTC m=+4283.725988519"
Nov 02 21:00:02.842584 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:02.842537    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv" podStartSLOduration=2.8425223429999997 podStartE2EDuration="2.842522343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:02.841488031 +0000 UTC m=+4283.839176640" watchObservedRunningTime="2026-11-02 21:00:02.842522343 +0000 UTC m=+4283.840210982"
Nov 02 21:00:03.181484 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.181451    5224 patch_prober.go:29] interesting pod/e2e-inf-b-801923-openshift-default-699785fb7-ts775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 21:00:03.181632 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.181503    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775" podUID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 21:00:03.295239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.295205    5224 generic.go:356] "Generic (PLEG): container finished" podID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122" containerID="28c8a6ab64330133af2cd5cee4f7a1bf95ffb26ef9901700356dc7eb2526ad9d" exitCode=60
Nov 02 21:00:03.295700 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.295285    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr" event={"ID":"b6d2d907-945a-4fd8-bb27-5a4c78b2b122","Type":"ContainerDied","Data":"28c8a6ab64330133af2cd5cee4f7a1bf95ffb26ef9901700356dc7eb2526ad9d"}
Nov 02 21:00:03.295700 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.295606    5224 scope.go:122] "RemoveContainer" containerID="28c8a6ab64330133af2cd5cee4f7a1bf95ffb26ef9901700356dc7eb2526ad9d"
Nov 02 21:00:03.296962 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.296938    5224 generic.go:356] "Generic (PLEG): container finished" podID="72cd7435-3e8f-4e42-8c96-08f4eafa8437" containerID="108acd7b7d2adb44d31071c077576b4da8556a3aa28d68d3eb3ccfd2bd00cf97" exitCode=60
Nov 02 21:00:03.297073 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.296972    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb" event={"ID":"72cd7435-3e8f-4e42-8c96-08f4eafa8437","Type":"ContainerDied","Data":"108acd7b7d2adb44d31071c077576b4da8556a3aa28d68d3eb3ccfd2bd00cf97"}
Nov 02 21:00:03.297235 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.297217    5224 scope.go:122] "RemoveContainer" containerID="108acd7b7d2adb44d31071c077576b4da8556a3aa28d68d3eb3ccfd2bd00cf97"
Nov 02 21:00:03.299149 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.299129    5224 generic.go:356] "Generic (PLEG): container finished" podID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581" containerID="d0d4bf96ca92add5b03327a38fed8e66028d4162b9dc4a94efc45dfdaf5ffc42" exitCode=60
Nov 02 21:00:03.299297 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.299232    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv" event={"ID":"3c3ebb63-cca2-47c5-be89-1dbfdd34c581","Type":"ContainerDied","Data":"d0d4bf96ca92add5b03327a38fed8e66028d4162b9dc4a94efc45dfdaf5ffc42"}
Nov 02 21:00:03.299429 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.299411    5224 scope.go:122] "RemoveContainer" containerID="d0d4bf96ca92add5b03327a38fed8e66028d4162b9dc4a94efc45dfdaf5ffc42"
Nov 02 21:00:03.301386 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.301367    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk" event={"ID":"89d66f8a-6db0-4cab-9c46-d28df83c9953","Type":"ContainerStarted","Data":"735eb6213c479088e83c1c4a4176485ef5113c8db18c95e19a7b1ef99d71cd66"}
Nov 02 21:00:03.302994 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.302975    5224 generic.go:356] "Generic (PLEG): container finished" podID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a" containerID="fb26b9e4f31694118f09d4d273544d7cbddbe41ea0c12eb36000bae123c83c01" exitCode=60
Nov 02 21:00:03.303074 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.303022    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5" event={"ID":"5a4383ae-73a5-45a1-979d-16a9dbb7e83a","Type":"ContainerDied","Data":"fb26b9e4f31694118f09d4d273544d7cbddbe41ea0c12eb36000bae123c83c01"}
Nov 02 21:00:03.303214 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.303200    5224 scope.go:122] "RemoveContainer" containerID="fb26b9e4f31694118f09d4d273544d7cbddbe41ea0c12eb36000bae123c83c01"
Nov 02 21:00:03.669792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:03.657907    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk" podStartSLOduration=3.657886806 podStartE2EDuration="3.657886806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:03.648857254 +0000 UTC m=+4284.646545864" watchObservedRunningTime="2026-11-02 21:00:03.657886806 +0000 UTC m=+4284.655575417"
Nov 02 21:00:04.182281 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:04.182186    5224 patch_prober.go:29] interesting pod/e2e-inf-b-801923-openshift-default-699785fb7-ts775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 21:00:04.182281 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:04.182245    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775" podUID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 21:00:04.312270 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:04.312226    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr" event={"ID":"b6d2d907-945a-4fd8-bb27-5a4c78b2b122","Type":"ContainerStarted","Data":"d7eeed4ab801c77c992fe878b241f12dfe67069a01932a7df2a2967d8316dd87"}
Nov 02 21:00:04.314388 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:04.314352    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb" event={"ID":"72cd7435-3e8f-4e42-8c96-08f4eafa8437","Type":"ContainerStarted","Data":"6f561a0a4671a2dbfc59699cfb32053be0219c0cf4bca5c065e8f973e35ebb8e"}
Nov 02 21:00:04.316328 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:04.316303    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv" event={"ID":"3c3ebb63-cca2-47c5-be89-1dbfdd34c581","Type":"ContainerStarted","Data":"ea0adbcef6169efbceb79efe288ec9af97927d56dcc5ece7a502a66000474001"}
Nov 02 21:00:04.318384 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:04.318361    5224 generic.go:356] "Generic (PLEG): container finished" podID="89d66f8a-6db0-4cab-9c46-d28df83c9953" containerID="735eb6213c479088e83c1c4a4176485ef5113c8db18c95e19a7b1ef99d71cd66" exitCode=60
Nov 02 21:00:04.318520 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:04.318417    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk" event={"ID":"89d66f8a-6db0-4cab-9c46-d28df83c9953","Type":"ContainerDied","Data":"735eb6213c479088e83c1c4a4176485ef5113c8db18c95e19a7b1ef99d71cd66"}
Nov 02 21:00:04.318520 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:04.318464    5224 scope.go:122] "RemoveContainer" containerID="0a32904987b456da17b091207c927b0e2a32d8667b8222b39196ccb42a6a0ce9"
Nov 02 21:00:04.318648 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:04.318638    5224 scope.go:122] "RemoveContainer" containerID="735eb6213c479088e83c1c4a4176485ef5113c8db18c95e19a7b1ef99d71cd66"
Nov 02 21:00:04.318948 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:04.318923    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk_odh-ai-gateway-infra(89d66f8a-6db0-4cab-9c46-d28df83c9953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk" podUID="89d66f8a-6db0-4cab-9c46-d28df83c9953"
Nov 02 21:00:04.320562 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:04.320543    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5" event={"ID":"5a4383ae-73a5-45a1-979d-16a9dbb7e83a","Type":"ContainerStarted","Data":"ce2e13657b3558359e5215832c6bdf69b882f623c7274d4aa3d3c5053edc831e"}
Nov 02 21:00:05.186296 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.186265    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 21:00:05.187372 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.187348    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 21:00:05.328089 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.328059    5224 scope.go:122] "RemoveContainer" containerID="735eb6213c479088e83c1c4a4176485ef5113c8db18c95e19a7b1ef99d71cd66"
Nov 02 21:00:05.328530 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:05.328352    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk_odh-ai-gateway-infra(89d66f8a-6db0-4cab-9c46-d28df83c9953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk" podUID="89d66f8a-6db0-4cab-9c46-d28df83c9953"
Nov 02 21:00:05.329711 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.329680    5224 generic.go:356] "Generic (PLEG): container finished" podID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a" containerID="ce2e13657b3558359e5215832c6bdf69b882f623c7274d4aa3d3c5053edc831e" exitCode=60
Nov 02 21:00:05.329840 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.329758    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5" event={"ID":"5a4383ae-73a5-45a1-979d-16a9dbb7e83a","Type":"ContainerDied","Data":"ce2e13657b3558359e5215832c6bdf69b882f623c7274d4aa3d3c5053edc831e"}
Nov 02 21:00:05.329840 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.329836    5224 scope.go:122] "RemoveContainer" containerID="fb26b9e4f31694118f09d4d273544d7cbddbe41ea0c12eb36000bae123c83c01"
Nov 02 21:00:05.329992 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.329974    5224 scope.go:122] "RemoveContainer" containerID="ce2e13657b3558359e5215832c6bdf69b882f623c7274d4aa3d3c5053edc831e"
Nov 02 21:00:05.330242 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:05.330221    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5_odh-ai-gateway-infra(5a4383ae-73a5-45a1-979d-16a9dbb7e83a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5" podUID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a"
Nov 02 21:00:05.331828 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.331805    5224 generic.go:356] "Generic (PLEG): container finished" podID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122" containerID="d7eeed4ab801c77c992fe878b241f12dfe67069a01932a7df2a2967d8316dd87" exitCode=60
Nov 02 21:00:05.331905 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.331889    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr" event={"ID":"b6d2d907-945a-4fd8-bb27-5a4c78b2b122","Type":"ContainerDied","Data":"d7eeed4ab801c77c992fe878b241f12dfe67069a01932a7df2a2967d8316dd87"}
Nov 02 21:00:05.332086 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.332072    5224 scope.go:122] "RemoveContainer" containerID="d7eeed4ab801c77c992fe878b241f12dfe67069a01932a7df2a2967d8316dd87"
Nov 02 21:00:05.332334 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:05.332311    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894220-rgdhr_odh-ai-gateway-infra(b6d2d907-945a-4fd8-bb27-5a4c78b2b122)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr" podUID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122"
Nov 02 21:00:05.333646 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.333627    5224 generic.go:356] "Generic (PLEG): container finished" podID="72cd7435-3e8f-4e42-8c96-08f4eafa8437" containerID="6f561a0a4671a2dbfc59699cfb32053be0219c0cf4bca5c065e8f973e35ebb8e" exitCode=60
Nov 02 21:00:05.333756 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.333709    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb" event={"ID":"72cd7435-3e8f-4e42-8c96-08f4eafa8437","Type":"ContainerDied","Data":"6f561a0a4671a2dbfc59699cfb32053be0219c0cf4bca5c065e8f973e35ebb8e"}
Nov 02 21:00:05.333913 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.333899    5224 scope.go:122] "RemoveContainer" containerID="6f561a0a4671a2dbfc59699cfb32053be0219c0cf4bca5c065e8f973e35ebb8e"
Nov 02 21:00:05.334163 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:05.334139    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb_odh-ai-gateway-infra(72cd7435-3e8f-4e42-8c96-08f4eafa8437)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb" podUID="72cd7435-3e8f-4e42-8c96-08f4eafa8437"
Nov 02 21:00:05.335544 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.335519    5224 generic.go:356] "Generic (PLEG): container finished" podID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581" containerID="ea0adbcef6169efbceb79efe288ec9af97927d56dcc5ece7a502a66000474001" exitCode=60
Nov 02 21:00:05.335637 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.335583    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv" event={"ID":"3c3ebb63-cca2-47c5-be89-1dbfdd34c581","Type":"ContainerDied","Data":"ea0adbcef6169efbceb79efe288ec9af97927d56dcc5ece7a502a66000474001"}
Nov 02 21:00:05.335761 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.335744    5224 scope.go:122] "RemoveContainer" containerID="ea0adbcef6169efbceb79efe288ec9af97927d56dcc5ece7a502a66000474001"
Nov 02 21:00:05.335978 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:05.335960    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv_odh-ai-gateway-infra(3c3ebb63-cca2-47c5-be89-1dbfdd34c581)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv" podUID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581"
Nov 02 21:00:05.342600 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.342583    5224 scope.go:122] "RemoveContainer" containerID="28c8a6ab64330133af2cd5cee4f7a1bf95ffb26ef9901700356dc7eb2526ad9d"
Nov 02 21:00:05.355413 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.355395    5224 scope.go:122] "RemoveContainer" containerID="108acd7b7d2adb44d31071c077576b4da8556a3aa28d68d3eb3ccfd2bd00cf97"
Nov 02 21:00:05.367566 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:05.367548    5224 scope.go:122] "RemoveContainer" containerID="d0d4bf96ca92add5b03327a38fed8e66028d4162b9dc4a94efc45dfdaf5ffc42"
Nov 02 21:00:06.343958 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:06.343927    5224 scope.go:122] "RemoveContainer" containerID="d7eeed4ab801c77c992fe878b241f12dfe67069a01932a7df2a2967d8316dd87"
Nov 02 21:00:06.344503 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:06.344268    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894220-rgdhr_odh-ai-gateway-infra(b6d2d907-945a-4fd8-bb27-5a4c78b2b122)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr" podUID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122"
Nov 02 21:00:06.345808 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:06.345761    5224 scope.go:122] "RemoveContainer" containerID="6f561a0a4671a2dbfc59699cfb32053be0219c0cf4bca5c065e8f973e35ebb8e"
Nov 02 21:00:06.346120 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:06.346097    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb_odh-ai-gateway-infra(72cd7435-3e8f-4e42-8c96-08f4eafa8437)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb" podUID="72cd7435-3e8f-4e42-8c96-08f4eafa8437"
Nov 02 21:00:06.347343 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:06.347319    5224 scope.go:122] "RemoveContainer" containerID="ea0adbcef6169efbceb79efe288ec9af97927d56dcc5ece7a502a66000474001"
Nov 02 21:00:06.347591 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:06.347572    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv_odh-ai-gateway-infra(3c3ebb63-cca2-47c5-be89-1dbfdd34c581)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv" podUID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581"
Nov 02 21:00:06.349098 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:06.349081    5224 scope.go:122] "RemoveContainer" containerID="ce2e13657b3558359e5215832c6bdf69b882f623c7274d4aa3d3c5053edc831e"
Nov 02 21:00:06.349299 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:06.349281    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5_odh-ai-gateway-infra(5a4383ae-73a5-45a1-979d-16a9dbb7e83a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5" podUID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a"
Nov 02 21:00:06.388329 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:06.388295    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:08.737030 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:08.736992    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:09.453943 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:09.453904    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:10.889457 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:10.889414    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:11.248956 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:11.248871    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:15.160922 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:15.160891    5224 scope.go:122] "RemoveContainer" containerID="d7eeed4ab801c77c992fe878b241f12dfe67069a01932a7df2a2967d8316dd87"
Nov 02 21:00:16.071577 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:16.071540    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:16.161348 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:16.161317    5224 scope.go:122] "RemoveContainer" containerID="735eb6213c479088e83c1c4a4176485ef5113c8db18c95e19a7b1ef99d71cd66"
Nov 02 21:00:16.413094 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:16.413056    5224 generic.go:356] "Generic (PLEG): container finished" podID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122" containerID="208698ebb183a750bf35d500ecc393495c248d94f4748aba4afd8c90aa11a16a" exitCode=60
Nov 02 21:00:16.413291 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:16.413125    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr" event={"ID":"b6d2d907-945a-4fd8-bb27-5a4c78b2b122","Type":"ContainerDied","Data":"208698ebb183a750bf35d500ecc393495c248d94f4748aba4afd8c90aa11a16a"}
Nov 02 21:00:16.413291 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:16.413191    5224 scope.go:122] "RemoveContainer" containerID="d7eeed4ab801c77c992fe878b241f12dfe67069a01932a7df2a2967d8316dd87"
Nov 02 21:00:16.413610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:16.413592    5224 scope.go:122] "RemoveContainer" containerID="208698ebb183a750bf35d500ecc393495c248d94f4748aba4afd8c90aa11a16a"
Nov 02 21:00:16.413952 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:16.413933    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894220-rgdhr_odh-ai-gateway-infra(b6d2d907-945a-4fd8-bb27-5a4c78b2b122)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr" podUID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122"
Nov 02 21:00:17.423450 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:17.423413    5224 generic.go:356] "Generic (PLEG): container finished" podID="89d66f8a-6db0-4cab-9c46-d28df83c9953" containerID="e9db14b1d7eaeb291b86c59918e886fff8135eff92952445029ebf164747c8f5" exitCode=60
Nov 02 21:00:17.423450 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:17.423449    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk" event={"ID":"89d66f8a-6db0-4cab-9c46-d28df83c9953","Type":"ContainerDied","Data":"e9db14b1d7eaeb291b86c59918e886fff8135eff92952445029ebf164747c8f5"}
Nov 02 21:00:17.424056 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:17.423512    5224 scope.go:122] "RemoveContainer" containerID="735eb6213c479088e83c1c4a4176485ef5113c8db18c95e19a7b1ef99d71cd66"
Nov 02 21:00:17.424056 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:17.423965    5224 scope.go:122] "RemoveContainer" containerID="e9db14b1d7eaeb291b86c59918e886fff8135eff92952445029ebf164747c8f5"
Nov 02 21:00:17.424319 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:17.424296    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk_odh-ai-gateway-infra(89d66f8a-6db0-4cab-9c46-d28df83c9953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk" podUID="89d66f8a-6db0-4cab-9c46-d28df83c9953"
Nov 02 21:00:17.538314 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:17.538220    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"]
Nov 02 21:00:17.691809 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:17.691725    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"
Nov 02 21:00:17.808792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:17.808722    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-maas-api-service-ca\") pod \"b6d2d907-945a-4fd8-bb27-5a4c78b2b122\" (UID: \"b6d2d907-945a-4fd8-bb27-5a4c78b2b122\") "
Nov 02 21:00:17.809042 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:17.809012    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-kube-api-access-m4gnv\" (UniqueName: \"kubernetes.io/projected/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-kube-api-access-m4gnv\") pod \"b6d2d907-945a-4fd8-bb27-5a4c78b2b122\" (UID: \"b6d2d907-945a-4fd8-bb27-5a4c78b2b122\") "
Nov 02 21:00:17.809300 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:17.809167    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-maas-api-service-ca" pod "b6d2d907-945a-4fd8-bb27-5a4c78b2b122" (UID: "b6d2d907-945a-4fd8-bb27-5a4c78b2b122"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:00:17.809796 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:17.809752    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:00:17.811814 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:17.811784    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-kube-api-access-m4gnv" pod "b6d2d907-945a-4fd8-bb27-5a4c78b2b122" (UID: "b6d2d907-945a-4fd8-bb27-5a4c78b2b122"). InnerVolumeSpecName "kube-api-access-m4gnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:00:17.911285 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:17.911245    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4gnv\" (UniqueName: \"kubernetes.io/projected/b6d2d907-945a-4fd8-bb27-5a4c78b2b122-kube-api-access-m4gnv\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:00:18.160528 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.160494    5224 scope.go:122] "RemoveContainer" containerID="6f561a0a4671a2dbfc59699cfb32053be0219c0cf4bca5c065e8f973e35ebb8e"
Nov 02 21:00:18.160725 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.160602    5224 scope.go:122] "RemoveContainer" containerID="ce2e13657b3558359e5215832c6bdf69b882f623c7274d4aa3d3c5053edc831e"
Nov 02 21:00:18.437252 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.437157    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"
Nov 02 21:00:18.437252 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.437178    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr" event={"ID":"b6d2d907-945a-4fd8-bb27-5a4c78b2b122","Type":"ContainerDied","Data":"ac9e70f242b455a0581556c13b2752d0dad5557dc1770682d284240a994c66ca"}
Nov 02 21:00:18.437812 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.437275    5224 scope.go:122] "RemoveContainer" containerID="208698ebb183a750bf35d500ecc393495c248d94f4748aba4afd8c90aa11a16a"
Nov 02 21:00:18.549711 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.549675    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"]
Nov 02 21:00:18.607433 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.607395    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"]
Nov 02 21:00:18.611229 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.611199    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-rgdhr"]
Nov 02 21:00:18.708641 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.708611    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"
Nov 02 21:00:18.821384 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.821346    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d66f8a-6db0-4cab-9c46-d28df83c9953-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d66f8a-6db0-4cab-9c46-d28df83c9953-maas-api-service-ca\") pod \"89d66f8a-6db0-4cab-9c46-d28df83c9953\" (UID: \"89d66f8a-6db0-4cab-9c46-d28df83c9953\") "
Nov 02 21:00:18.821581 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.821542    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d66f8a-6db0-4cab-9c46-d28df83c9953-kube-api-access-5xl5c\" (UniqueName: \"kubernetes.io/projected/89d66f8a-6db0-4cab-9c46-d28df83c9953-kube-api-access-5xl5c\") pod \"89d66f8a-6db0-4cab-9c46-d28df83c9953\" (UID: \"89d66f8a-6db0-4cab-9c46-d28df83c9953\") "
Nov 02 21:00:18.821840 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.821760    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d66f8a-6db0-4cab-9c46-d28df83c9953-maas-api-service-ca" pod "89d66f8a-6db0-4cab-9c46-d28df83c9953" (UID: "89d66f8a-6db0-4cab-9c46-d28df83c9953"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:00:18.822118 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.822099    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d66f8a-6db0-4cab-9c46-d28df83c9953-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:00:18.823676 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.823645    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d66f8a-6db0-4cab-9c46-d28df83c9953-kube-api-access-5xl5c" pod "89d66f8a-6db0-4cab-9c46-d28df83c9953" (UID: "89d66f8a-6db0-4cab-9c46-d28df83c9953"). InnerVolumeSpecName "kube-api-access-5xl5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:00:18.923536 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:18.923498    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xl5c\" (UniqueName: \"kubernetes.io/projected/89d66f8a-6db0-4cab-9c46-d28df83c9953-kube-api-access-5xl5c\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:00:19.165288 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.165251    5224 scope.go:122] "RemoveContainer" containerID="ea0adbcef6169efbceb79efe288ec9af97927d56dcc5ece7a502a66000474001"
Nov 02 21:00:19.169395 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.169373    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122" path="/var/lib/kubelet/pods/b6d2d907-945a-4fd8-bb27-5a4c78b2b122/volumes"
Nov 02 21:00:19.448504 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.448415    5224 generic.go:356] "Generic (PLEG): container finished" podID="72cd7435-3e8f-4e42-8c96-08f4eafa8437" containerID="19cb1816cd2ab4534e0f6e751f6b55267a61c61ea729bbf1295a1b8d90f7290e" exitCode=60
Nov 02 21:00:19.448504 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.448487    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb" event={"ID":"72cd7435-3e8f-4e42-8c96-08f4eafa8437","Type":"ContainerDied","Data":"19cb1816cd2ab4534e0f6e751f6b55267a61c61ea729bbf1295a1b8d90f7290e"}
Nov 02 21:00:19.449060 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.448556    5224 scope.go:122] "RemoveContainer" containerID="6f561a0a4671a2dbfc59699cfb32053be0219c0cf4bca5c065e8f973e35ebb8e"
Nov 02 21:00:19.449060 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.448991    5224 scope.go:122] "RemoveContainer" containerID="19cb1816cd2ab4534e0f6e751f6b55267a61c61ea729bbf1295a1b8d90f7290e"
Nov 02 21:00:19.449393 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:19.449355    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb_odh-ai-gateway-infra(72cd7435-3e8f-4e42-8c96-08f4eafa8437)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb" podUID="72cd7435-3e8f-4e42-8c96-08f4eafa8437"
Nov 02 21:00:19.450047 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.450030    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"
Nov 02 21:00:19.450157 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.450102    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk" event={"ID":"89d66f8a-6db0-4cab-9c46-d28df83c9953","Type":"ContainerDied","Data":"a0e90505c2fe2f8c25a021652ae5d0b9dc06aac97468ec3caa10bb264db931cc"}
Nov 02 21:00:19.452336 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.452317    5224 generic.go:356] "Generic (PLEG): container finished" podID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a" containerID="91638fdf753514684c19da88410aac4b3b707827c95e7e8e03048e6b23086d65" exitCode=60
Nov 02 21:00:19.452428 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.452390    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5" event={"ID":"5a4383ae-73a5-45a1-979d-16a9dbb7e83a","Type":"ContainerDied","Data":"91638fdf753514684c19da88410aac4b3b707827c95e7e8e03048e6b23086d65"}
Nov 02 21:00:19.452855 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.452837    5224 scope.go:122] "RemoveContainer" containerID="91638fdf753514684c19da88410aac4b3b707827c95e7e8e03048e6b23086d65"
Nov 02 21:00:19.453170 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:19.453145    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5_odh-ai-gateway-infra(5a4383ae-73a5-45a1-979d-16a9dbb7e83a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5" podUID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a"
Nov 02 21:00:19.473401 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.473373    5224 scope.go:122] "RemoveContainer" containerID="e9db14b1d7eaeb291b86c59918e886fff8135eff92952445029ebf164747c8f5"
Nov 02 21:00:19.482807 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.482782    5224 scope.go:122] "RemoveContainer" containerID="ce2e13657b3558359e5215832c6bdf69b882f623c7274d4aa3d3c5053edc831e"
Nov 02 21:00:19.790490 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.790451    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"]
Nov 02 21:00:19.793566 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:19.793537    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f97409-29894220-tqrwk"]
Nov 02 21:00:20.464761 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.464662    5224 generic.go:356] "Generic (PLEG): container finished" podID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581" containerID="06f45d9d2f7a854642ee20894d56041905b01b37d52b4e51ccee9c08fb7ba030" exitCode=60
Nov 02 21:00:20.464761 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.464723    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv" event={"ID":"3c3ebb63-cca2-47c5-be89-1dbfdd34c581","Type":"ContainerDied","Data":"06f45d9d2f7a854642ee20894d56041905b01b37d52b4e51ccee9c08fb7ba030"}
Nov 02 21:00:20.465281 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.464793    5224 scope.go:122] "RemoveContainer" containerID="ea0adbcef6169efbceb79efe288ec9af97927d56dcc5ece7a502a66000474001"
Nov 02 21:00:20.465281 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.465250    5224 scope.go:122] "RemoveContainer" containerID="06f45d9d2f7a854642ee20894d56041905b01b37d52b4e51ccee9c08fb7ba030"
Nov 02 21:00:20.465631 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:20.465605    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv_odh-ai-gateway-infra(3c3ebb63-cca2-47c5-be89-1dbfdd34c581)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv" podUID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581"
Nov 02 21:00:20.577422 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.577387    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"]
Nov 02 21:00:20.687009 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.686977    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"]
Nov 02 21:00:20.738942 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.738910    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"
Nov 02 21:00:20.846568 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.846530    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72cd7435-3e8f-4e42-8c96-08f4eafa8437-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cd7435-3e8f-4e42-8c96-08f4eafa8437-maas-api-service-ca\") pod \"72cd7435-3e8f-4e42-8c96-08f4eafa8437\" (UID: \"72cd7435-3e8f-4e42-8c96-08f4eafa8437\") "
Nov 02 21:00:20.846727 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.846584    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cd7435-3e8f-4e42-8c96-08f4eafa8437-kube-api-access-8pkxl\" (UniqueName: \"kubernetes.io/projected/72cd7435-3e8f-4e42-8c96-08f4eafa8437-kube-api-access-8pkxl\") pod \"72cd7435-3e8f-4e42-8c96-08f4eafa8437\" (UID: \"72cd7435-3e8f-4e42-8c96-08f4eafa8437\") "
Nov 02 21:00:20.846936 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.846913    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72cd7435-3e8f-4e42-8c96-08f4eafa8437-maas-api-service-ca" pod "72cd7435-3e8f-4e42-8c96-08f4eafa8437" (UID: "72cd7435-3e8f-4e42-8c96-08f4eafa8437"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:00:20.847136 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.847117    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cd7435-3e8f-4e42-8c96-08f4eafa8437-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:00:20.848992 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.848968    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cd7435-3e8f-4e42-8c96-08f4eafa8437-kube-api-access-8pkxl" pod "72cd7435-3e8f-4e42-8c96-08f4eafa8437" (UID: "72cd7435-3e8f-4e42-8c96-08f4eafa8437"). InnerVolumeSpecName "kube-api-access-8pkxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:00:20.874821 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.874793    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"
Nov 02 21:00:20.896579 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.896542    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:20.947681 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.947642    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-kube-api-access-9lf9r\" (UniqueName: \"kubernetes.io/projected/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-kube-api-access-9lf9r\") pod \"5a4383ae-73a5-45a1-979d-16a9dbb7e83a\" (UID: \"5a4383ae-73a5-45a1-979d-16a9dbb7e83a\") "
Nov 02 21:00:20.947926 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.947863    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-maas-api-service-ca\") pod \"5a4383ae-73a5-45a1-979d-16a9dbb7e83a\" (UID: \"5a4383ae-73a5-45a1-979d-16a9dbb7e83a\") "
Nov 02 21:00:20.948297 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.948269    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-maas-api-service-ca" pod "5a4383ae-73a5-45a1-979d-16a9dbb7e83a" (UID: "5a4383ae-73a5-45a1-979d-16a9dbb7e83a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:00:20.948466 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.948446    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:00:20.948544 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.948472    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pkxl\" (UniqueName: \"kubernetes.io/projected/72cd7435-3e8f-4e42-8c96-08f4eafa8437-kube-api-access-8pkxl\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:00:20.949870 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:20.949845    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-kube-api-access-9lf9r" pod "5a4383ae-73a5-45a1-979d-16a9dbb7e83a" (UID: "5a4383ae-73a5-45a1-979d-16a9dbb7e83a"). InnerVolumeSpecName "kube-api-access-9lf9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:00:21.051640 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.051606    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lf9r\" (UniqueName: \"kubernetes.io/projected/5a4383ae-73a5-45a1-979d-16a9dbb7e83a-kube-api-access-9lf9r\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:00:21.168364 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.168329    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d66f8a-6db0-4cab-9c46-d28df83c9953" path="/var/lib/kubelet/pods/89d66f8a-6db0-4cab-9c46-d28df83c9953/volumes"
Nov 02 21:00:21.472489 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.472406    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"
Nov 02 21:00:21.472489 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.472401    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb" event={"ID":"72cd7435-3e8f-4e42-8c96-08f4eafa8437","Type":"ContainerDied","Data":"4c21da503e7d6a559b07da7e94a10cd673629251948ba4af1d41c3f75513eace"}
Nov 02 21:00:21.473027 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.472527    5224 scope.go:122] "RemoveContainer" containerID="19cb1816cd2ab4534e0f6e751f6b55267a61c61ea729bbf1295a1b8d90f7290e"
Nov 02 21:00:21.476181 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.476158    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"
Nov 02 21:00:21.476181 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.476170    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5" event={"ID":"5a4383ae-73a5-45a1-979d-16a9dbb7e83a","Type":"ContainerDied","Data":"457d3224ca6af5a3af154fdfd5dd7fdb97b4ae735f2841675cde4aa2a79b58e0"}
Nov 02 21:00:21.483647 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.483613    5224 scope.go:122] "RemoveContainer" containerID="91638fdf753514684c19da88410aac4b3b707827c95e7e8e03048e6b23086d65"
Nov 02 21:00:21.594801 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.594736    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"]
Nov 02 21:00:21.643460 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.643426    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"]
Nov 02 21:00:21.647017 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.646994    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894220-8ngt5"]
Nov 02 21:00:21.759143 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.759112    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"]
Nov 02 21:00:21.760676 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.760624    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"
Nov 02 21:00:21.768072 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.768046    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e19d40-29894220-9lmmb"]
Nov 02 21:00:21.864022 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.863982    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-maas-api-service-ca\") pod \"3c3ebb63-cca2-47c5-be89-1dbfdd34c581\" (UID: \"3c3ebb63-cca2-47c5-be89-1dbfdd34c581\") "
Nov 02 21:00:21.864213 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.864139    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-kube-api-access-frd9b\" (UniqueName: \"kubernetes.io/projected/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-kube-api-access-frd9b\") pod \"3c3ebb63-cca2-47c5-be89-1dbfdd34c581\" (UID: \"3c3ebb63-cca2-47c5-be89-1dbfdd34c581\") "
Nov 02 21:00:21.864440 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.864418    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-maas-api-service-ca" pod "3c3ebb63-cca2-47c5-be89-1dbfdd34c581" (UID: "3c3ebb63-cca2-47c5-be89-1dbfdd34c581"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:00:21.864613 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.864592    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:00:21.866270 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.866243    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-kube-api-access-frd9b" pod "3c3ebb63-cca2-47c5-be89-1dbfdd34c581" (UID: "3c3ebb63-cca2-47c5-be89-1dbfdd34c581"). InnerVolumeSpecName "kube-api-access-frd9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:00:21.965881 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:21.965841    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frd9b\" (UniqueName: \"kubernetes.io/projected/3c3ebb63-cca2-47c5-be89-1dbfdd34c581-kube-api-access-frd9b\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:00:22.033281 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:22.033242    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:00:22.488613 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:22.488512    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv" event={"ID":"3c3ebb63-cca2-47c5-be89-1dbfdd34c581","Type":"ContainerDied","Data":"e3fcda57c08585b113d60d4991b6860474a0c3f6248d53575a4b3249791e7766"}
Nov 02 21:00:22.488613 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:22.488552    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"
Nov 02 21:00:22.488613 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:22.488593    5224 scope.go:122] "RemoveContainer" containerID="06f45d9d2f7a854642ee20894d56041905b01b37d52b4e51ccee9c08fb7ba030"
Nov 02 21:00:22.657121 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:22.657075    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"]
Nov 02 21:00:22.708550 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:22.708510    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894220-2ncrv"]
Nov 02 21:00:23.169298 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:23.169263    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581" path="/var/lib/kubelet/pods/3c3ebb63-cca2-47c5-be89-1dbfdd34c581/volumes"
Nov 02 21:00:23.169989 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:23.169959    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a" path="/var/lib/kubelet/pods/5a4383ae-73a5-45a1-979d-16a9dbb7e83a/volumes"
Nov 02 21:00:23.170414 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:23.170394    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72cd7435-3e8f-4e42-8c96-08f4eafa8437" path="/var/lib/kubelet/pods/72cd7435-3e8f-4e42-8c96-08f4eafa8437/volumes"
Nov 02 21:00:25.986380 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:25.986348    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:27.359902 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:27.359864    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:00:30.004907 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:30.004865    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:30.878190 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:30.878150    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:34.476099 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:34.476060    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:34.644448 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:34.644406    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:35.683973 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:35.683943    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:35.857544 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:35.857506    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:37.696341 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:37.696291    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:37.904385 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:37.904347    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:38.080172 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:38.080137    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:41.197011 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:41.196914    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:41.364252 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:41.364214    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:43.912268 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:43.912227    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:44.804667 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:44.804614    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:45.021174 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:45.021138    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:45.707534 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:45.707493    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:46.453038 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:46.452997    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:49.709622 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:49.709545    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:51.768934 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:51.768899    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:56.650719 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:00:56.650685    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:00:57.359267 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:00:57.359230    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:01:01.487071 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:01.487035    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:05.052833 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:05.052792    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:05.995663 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:05.995631    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:06.213366 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:06.213328    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:06.444131 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:06.444098    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:08.409039 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:08.409005    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:08.633286 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:08.633252    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:09.092055 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:09.092020    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:09.873465 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:09.873432    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:10.757327 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:10.757288    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:11.395402 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:11.395360    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:11.779817 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:11.779737    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:14.573569 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:14.573535    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:14.782880 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:14.782841    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:15.585077 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:15.585044    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:16.401227 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:16.401187    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:20.754926 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:20.754891    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:20.928983 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:20.928944    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:21.405180 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:21.405144    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:22.434644 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:22.434610    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:26.278064 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:26.278025    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:27.310780 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:27.310730    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:27.361616 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:01:27.361577    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:01:27.594458 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:27.594368    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:29.129313 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:29.129277    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:29.312466 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:29.312420    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:31.152464 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:31.152428    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:35.962037 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:35.961996    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:40.789168 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:40.789122    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:44.142258 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:44.142223    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:01:44.898257 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:44.898219    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"]
Nov 02 21:01:44.898530 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:44.898425    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs" podUID="972297c3-0639-4570-8636-5d29249c866f" containerName="maas-api" containerID="cri-o://234a879d1e4f112b955a10629cf8e6875bc21bc4b686b41148f68b0e9ca45e67" gracePeriod=30
Nov 02 21:01:45.049276 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.049235    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:45.051427 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.051398    5224 generic.go:356] "Generic (PLEG): container finished" podID="972297c3-0639-4570-8636-5d29249c866f" containerID="234a879d1e4f112b955a10629cf8e6875bc21bc4b686b41148f68b0e9ca45e67" exitCode=0
Nov 02 21:01:45.051576 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.051476    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs" event={"ID":"972297c3-0639-4570-8636-5d29249c866f","Type":"ContainerDied","Data":"234a879d1e4f112b955a10629cf8e6875bc21bc4b686b41148f68b0e9ca45e67"}
Nov 02 21:01:45.172265 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.172181    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 21:01:45.304138 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.304096    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-maas-api-tls\") pod \"972297c3-0639-4570-8636-5d29249c866f\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") "
Nov 02 21:01:45.304603 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.304581    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-metrics-certs\") pod \"972297c3-0639-4570-8636-5d29249c866f\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") "
Nov 02 21:01:45.304793 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.304758    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972297c3-0639-4570-8636-5d29249c866f-kube-api-access-sw5ch\" (UniqueName: \"kubernetes.io/projected/972297c3-0639-4570-8636-5d29249c866f-kube-api-access-sw5ch\") pod \"972297c3-0639-4570-8636-5d29249c866f\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") "
Nov 02 21:01:45.304976 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.304955    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/972297c3-0639-4570-8636-5d29249c866f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972297c3-0639-4570-8636-5d29249c866f-openshift-service-ca\") pod \"972297c3-0639-4570-8636-5d29249c866f\" (UID: \"972297c3-0639-4570-8636-5d29249c866f\") "
Nov 02 21:01:45.307250 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.307222    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972297c3-0639-4570-8636-5d29249c866f-openshift-service-ca" pod "972297c3-0639-4570-8636-5d29249c866f" (UID: "972297c3-0639-4570-8636-5d29249c866f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:01:45.308926 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.308888    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-maas-api-tls" pod "972297c3-0639-4570-8636-5d29249c866f" (UID: "972297c3-0639-4570-8636-5d29249c866f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:01:45.317637 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.317022    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-metrics-certs" pod "972297c3-0639-4570-8636-5d29249c866f" (UID: "972297c3-0639-4570-8636-5d29249c866f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:01:45.339265 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.339221    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972297c3-0639-4570-8636-5d29249c866f-kube-api-access-sw5ch" pod "972297c3-0639-4570-8636-5d29249c866f" (UID: "972297c3-0639-4570-8636-5d29249c866f"). InnerVolumeSpecName "kube-api-access-sw5ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:01:45.407316 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.406955    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:01:45.407316 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.407030    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw5ch\" (UniqueName: \"kubernetes.io/projected/972297c3-0639-4570-8636-5d29249c866f-kube-api-access-sw5ch\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:01:45.407316 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.407053    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972297c3-0639-4570-8636-5d29249c866f-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:01:45.407316 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.407069    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972297c3-0639-4570-8636-5d29249c866f-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:01:45.741816 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:45.741756    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:46.061947 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:46.061906    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs" event={"ID":"972297c3-0639-4570-8636-5d29249c866f","Type":"ContainerDied","Data":"780dde27af28c8f4ca9ee77a09f374d34f140e6e9e192c61977c4b5ce134af91"}
Nov 02 21:01:46.062218 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:46.062009    5224 scope.go:122] "RemoveContainer" containerID="234a879d1e4f112b955a10629cf8e6875bc21bc4b686b41148f68b0e9ca45e67"
Nov 02 21:01:46.062218 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:46.061926    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"
Nov 02 21:01:46.249689 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:46.249635    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"]
Nov 02 21:01:46.317026 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:46.316922    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f97409-7ffd65f4f7-ffrzs"]
Nov 02 21:01:46.904946 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:46.904901    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:47.222405 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:47.222298    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="972297c3-0639-4570-8636-5d29249c866f" path="/var/lib/kubelet/pods/972297c3-0639-4570-8636-5d29249c866f/volumes"
Nov 02 21:01:48.753966 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:48.753929    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:48.980617 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:48.980584    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:50.334335 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:50.334282    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:50.922732 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:50.922689    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:51.304653 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:51.304608    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:51.655496 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:51.655405    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:52.079254 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:52.079207    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:53.283912 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:53.283876    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:57.088900 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:57.088861    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:57.290961 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:57.290925    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:57.360809 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:01:57.360709    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:01:57.818315 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:57.818282    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:58.466395 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:58.466360    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:58.693192 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:58.693152    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:59.252708 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:59.252674    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:01:59.911970 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:01:59.911933    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:03.451660 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:03.451621    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:03.767546 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:03.767510    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:05.747264 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:05.747221    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:06.708553 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:06.708520    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:06.888486 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:06.888441    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:08.102627 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.102598    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"]
Nov 02 21:02:08.103066 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.102848    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn" podUID="dffa4399-a7ef-4f19-8177-42c954bfbc40" containerName="maas-api" containerID="cri-o://23f5de4e5362a3d161a0cd5fcebe84c0a4b750ae25104befa3beec6a4ccd9229" gracePeriod=30
Nov 02 21:02:08.242081 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.242035    5224 generic.go:356] "Generic (PLEG): container finished" podID="dffa4399-a7ef-4f19-8177-42c954bfbc40" containerID="23f5de4e5362a3d161a0cd5fcebe84c0a4b750ae25104befa3beec6a4ccd9229" exitCode=0
Nov 02 21:02:08.242238 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.242107    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn" event={"ID":"dffa4399-a7ef-4f19-8177-42c954bfbc40","Type":"ContainerDied","Data":"23f5de4e5362a3d161a0cd5fcebe84c0a4b750ae25104befa3beec6a4ccd9229"}
Nov 02 21:02:08.245834 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.245803    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:08.395394 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.395364    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 21:02:08.495870 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.495835    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dffa4399-a7ef-4f19-8177-42c954bfbc40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffa4399-a7ef-4f19-8177-42c954bfbc40-openshift-service-ca\") pod \"dffa4399-a7ef-4f19-8177-42c954bfbc40\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") "
Nov 02 21:02:08.496068 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.495896    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-maas-api-tls\") pod \"dffa4399-a7ef-4f19-8177-42c954bfbc40\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") "
Nov 02 21:02:08.496068 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.495929    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-metrics-certs\") pod \"dffa4399-a7ef-4f19-8177-42c954bfbc40\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") "
Nov 02 21:02:08.496068 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.496058    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffa4399-a7ef-4f19-8177-42c954bfbc40-kube-api-access-c4hhw\" (UniqueName: \"kubernetes.io/projected/dffa4399-a7ef-4f19-8177-42c954bfbc40-kube-api-access-c4hhw\") pod \"dffa4399-a7ef-4f19-8177-42c954bfbc40\" (UID: \"dffa4399-a7ef-4f19-8177-42c954bfbc40\") "
Nov 02 21:02:08.496312 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.496274    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffa4399-a7ef-4f19-8177-42c954bfbc40-openshift-service-ca" pod "dffa4399-a7ef-4f19-8177-42c954bfbc40" (UID: "dffa4399-a7ef-4f19-8177-42c954bfbc40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:02:08.496628 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.496602    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffa4399-a7ef-4f19-8177-42c954bfbc40-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:02:08.498191 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.498160    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-metrics-certs" pod "dffa4399-a7ef-4f19-8177-42c954bfbc40" (UID: "dffa4399-a7ef-4f19-8177-42c954bfbc40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:02:08.498295 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.498249    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffa4399-a7ef-4f19-8177-42c954bfbc40-kube-api-access-c4hhw" pod "dffa4399-a7ef-4f19-8177-42c954bfbc40" (UID: "dffa4399-a7ef-4f19-8177-42c954bfbc40"). InnerVolumeSpecName "kube-api-access-c4hhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:02:08.498353 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.498335    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-maas-api-tls" pod "dffa4399-a7ef-4f19-8177-42c954bfbc40" (UID: "dffa4399-a7ef-4f19-8177-42c954bfbc40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:02:08.598340 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.598304    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4hhw\" (UniqueName: \"kubernetes.io/projected/dffa4399-a7ef-4f19-8177-42c954bfbc40-kube-api-access-c4hhw\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:02:08.598340 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.598333    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:02:08.598340 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:08.598344    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffa4399-a7ef-4f19-8177-42c954bfbc40-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:02:09.249458 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:09.249430    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"
Nov 02 21:02:09.249882 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:09.249467    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn" event={"ID":"dffa4399-a7ef-4f19-8177-42c954bfbc40","Type":"ContainerDied","Data":"8f87ce01c05dc4059a57aae1c73994565a7b6db4094051e248658ec6060b0cba"}
Nov 02 21:02:09.249882 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:09.249527    5224 scope.go:122] "RemoveContainer" containerID="23f5de4e5362a3d161a0cd5fcebe84c0a4b750ae25104befa3beec6a4ccd9229"
Nov 02 21:02:09.417667 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:09.417635    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"]
Nov 02 21:02:09.459843 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:09.459741    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:09.472993 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:09.470621    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4530c5-6fb7c9d95b-qwlfn"]
Nov 02 21:02:10.248659 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:10.248609    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:10.426056 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:10.426017    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:10.710985 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:10.710936    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:10.898189 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:10.898149    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:11.167436 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:11.167403    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dffa4399-a7ef-4f19-8177-42c954bfbc40" path="/var/lib/kubelet/pods/dffa4399-a7ef-4f19-8177-42c954bfbc40/volumes"
Nov 02 21:02:11.391456 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:11.391423    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:12.058752 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:12.058716    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:12.281750 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:12.281714    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:13.032977 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:13.032933    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:13.211043 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:13.211006    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:14.255940 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:14.255898    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:14.458958 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:14.458923    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:15.864980 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:15.864940    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:20.766599 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:20.766521    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:25.675390 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:25.675343    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:27.045051 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:27.044995    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:27.357929 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:02:27.357850    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:02:30.479254 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:30.479077    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:32.656344 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:32.656303    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:35.358053 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:35.358017    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:35.596222 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:35.596187    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:35.889267 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:35.889232    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:36.476579 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:36.476544    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:40.259402 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:40.259364    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:41.648409 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:41.648373    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:42.044890 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:42.044856    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:02:45.275345 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:45.275305    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:50.144282 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:50.144248    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:54.964445 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:54.964396    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:02:57.358904 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:02:57.358870    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:02:59.742231 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:02:59.742191    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:02.792164 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:02.792126    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:04.581708 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:04.581668    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:08.396425 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:08.396378    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:09.385050 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:09.384988    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:09.795697 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:09.795662    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:10.518361 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:10.518318    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:13.459408 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:13.459369    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"]
Nov 02 21:03:13.601428 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:13.601388    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:13.651362 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:13.651305    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc" podUID="5840d9e3-7b79-408a-ab57-2dd8ca622771" containerName="maas-api" containerID="cri-o://332efa168b25a1c9ce1aa60177b2df5195958607eeeae830f4c5568db4185866" gracePeriod=30
Nov 02 21:03:13.806981 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:13.806939    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc" podUID="5840d9e3-7b79-408a-ab57-2dd8ca622771" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.176:8443/health\": dial tcp 10.134.0.176:8443: connect: connection refused"
Nov 02 21:03:13.979698 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:13.979674    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 21:03:14.054999 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.054927    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5840d9e3-7b79-408a-ab57-2dd8ca622771-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5840d9e3-7b79-408a-ab57-2dd8ca622771-openshift-service-ca\") pod \"5840d9e3-7b79-408a-ab57-2dd8ca622771\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") "
Nov 02 21:03:14.055188 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.055014    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-maas-api-tls\") pod \"5840d9e3-7b79-408a-ab57-2dd8ca622771\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") "
Nov 02 21:03:14.055188 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.055059    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-metrics-certs\") pod \"5840d9e3-7b79-408a-ab57-2dd8ca622771\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") "
Nov 02 21:03:14.055188 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.055142    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5840d9e3-7b79-408a-ab57-2dd8ca622771-kube-api-access-4jtwk\" (UniqueName: \"kubernetes.io/projected/5840d9e3-7b79-408a-ab57-2dd8ca622771-kube-api-access-4jtwk\") pod \"5840d9e3-7b79-408a-ab57-2dd8ca622771\" (UID: \"5840d9e3-7b79-408a-ab57-2dd8ca622771\") "
Nov 02 21:03:14.055382 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.055289    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5840d9e3-7b79-408a-ab57-2dd8ca622771-openshift-service-ca" pod "5840d9e3-7b79-408a-ab57-2dd8ca622771" (UID: "5840d9e3-7b79-408a-ab57-2dd8ca622771"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:03:14.055495 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.055478    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5840d9e3-7b79-408a-ab57-2dd8ca622771-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:14.057225 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.057194    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-maas-api-tls" pod "5840d9e3-7b79-408a-ab57-2dd8ca622771" (UID: "5840d9e3-7b79-408a-ab57-2dd8ca622771"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:03:14.057353 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.057231    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-metrics-certs" pod "5840d9e3-7b79-408a-ab57-2dd8ca622771" (UID: "5840d9e3-7b79-408a-ab57-2dd8ca622771"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:03:14.057353 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.057235    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5840d9e3-7b79-408a-ab57-2dd8ca622771-kube-api-access-4jtwk" pod "5840d9e3-7b79-408a-ab57-2dd8ca622771" (UID: "5840d9e3-7b79-408a-ab57-2dd8ca622771"). InnerVolumeSpecName "kube-api-access-4jtwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:03:14.156122 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.156086    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jtwk\" (UniqueName: \"kubernetes.io/projected/5840d9e3-7b79-408a-ab57-2dd8ca622771-kube-api-access-4jtwk\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:14.156122 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.156115    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:14.156122 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.156129    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5840d9e3-7b79-408a-ab57-2dd8ca622771-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:14.658611 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.658576    5224 generic.go:356] "Generic (PLEG): container finished" podID="5840d9e3-7b79-408a-ab57-2dd8ca622771" containerID="332efa168b25a1c9ce1aa60177b2df5195958607eeeae830f4c5568db4185866" exitCode=0
Nov 02 21:03:14.659067 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.658624    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"
Nov 02 21:03:14.659067 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.658664    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc" event={"ID":"5840d9e3-7b79-408a-ab57-2dd8ca622771","Type":"ContainerDied","Data":"332efa168b25a1c9ce1aa60177b2df5195958607eeeae830f4c5568db4185866"}
Nov 02 21:03:14.659067 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.658720    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc" event={"ID":"5840d9e3-7b79-408a-ab57-2dd8ca622771","Type":"ContainerDied","Data":"956932525e61fbfe80d30b34a86173a85316c9f1a463e9862659933331708f6b"}
Nov 02 21:03:14.659067 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.658748    5224 scope.go:122] "RemoveContainer" containerID="332efa168b25a1c9ce1aa60177b2df5195958607eeeae830f4c5568db4185866"
Nov 02 21:03:14.668977 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.668958    5224 scope.go:122] "RemoveContainer" containerID="332efa168b25a1c9ce1aa60177b2df5195958607eeeae830f4c5568db4185866"
Nov 02 21:03:14.669244 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:03:14.669221    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332efa168b25a1c9ce1aa60177b2df5195958607eeeae830f4c5568db4185866\": container with ID starting with 332efa168b25a1c9ce1aa60177b2df5195958607eeeae830f4c5568db4185866 not found: ID does not exist" containerID="332efa168b25a1c9ce1aa60177b2df5195958607eeeae830f4c5568db4185866"
Nov 02 21:03:14.669295 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.669255    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332efa168b25a1c9ce1aa60177b2df5195958607eeeae830f4c5568db4185866"} err="failed to get container status \"332efa168b25a1c9ce1aa60177b2df5195958607eeeae830f4c5568db4185866\": rpc error: code = NotFound desc = could not find container \"332efa168b25a1c9ce1aa60177b2df5195958607eeeae830f4c5568db4185866\": container with ID starting with 332efa168b25a1c9ce1aa60177b2df5195958607eeeae830f4c5568db4185866 not found: ID does not exist"
Nov 02 21:03:14.825558 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.825520    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"]
Nov 02 21:03:14.876473 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:14.876439    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e19d40-56f9867fb4-8hfdc"]
Nov 02 21:03:15.167234 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:15.167193    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5840d9e3-7b79-408a-ab57-2dd8ca622771" path="/var/lib/kubelet/pods/5840d9e3-7b79-408a-ab57-2dd8ca622771/volumes"
Nov 02 21:03:15.462548 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:15.462455    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:15.798986 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:15.798951    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:16.099053 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:16.098968    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:18.217019 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:18.216984    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:19.738663 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:19.738632    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"]
Nov 02 21:03:19.739151 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:19.738893    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw" podUID="13c4fdb1-f36b-4674-91b6-e84a17df72ce" containerName="maas-api" containerID="cri-o://e6534369605532efe0d64c46f51d2899282fd77ee5e4dd0bdc226cb4e706540d" gracePeriod=30
Nov 02 21:03:20.029552 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.029515    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 21:03:20.052992 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.052902    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:20.122506 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.122474    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-metrics-certs\") pod \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") "
Nov 02 21:03:20.122699 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.122520    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c4fdb1-f36b-4674-91b6-e84a17df72ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c4fdb1-f36b-4674-91b6-e84a17df72ce-openshift-service-ca\") pod \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") "
Nov 02 21:03:20.122742 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.122698    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c4fdb1-f36b-4674-91b6-e84a17df72ce-kube-api-access-wlvnn\" (UniqueName: \"kubernetes.io/projected/13c4fdb1-f36b-4674-91b6-e84a17df72ce-kube-api-access-wlvnn\") pod \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") "
Nov 02 21:03:20.122805 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.122751    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-maas-api-tls\") pod \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\" (UID: \"13c4fdb1-f36b-4674-91b6-e84a17df72ce\") "
Nov 02 21:03:20.123244 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.123214    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c4fdb1-f36b-4674-91b6-e84a17df72ce-openshift-service-ca" pod "13c4fdb1-f36b-4674-91b6-e84a17df72ce" (UID: "13c4fdb1-f36b-4674-91b6-e84a17df72ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:03:20.124889 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.124863    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-metrics-certs" pod "13c4fdb1-f36b-4674-91b6-e84a17df72ce" (UID: "13c4fdb1-f36b-4674-91b6-e84a17df72ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:03:20.125277 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.125257    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-maas-api-tls" pod "13c4fdb1-f36b-4674-91b6-e84a17df72ce" (UID: "13c4fdb1-f36b-4674-91b6-e84a17df72ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:03:20.125367 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.125297    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c4fdb1-f36b-4674-91b6-e84a17df72ce-kube-api-access-wlvnn" pod "13c4fdb1-f36b-4674-91b6-e84a17df72ce" (UID: "13c4fdb1-f36b-4674-91b6-e84a17df72ce"). InnerVolumeSpecName "kube-api-access-wlvnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:03:20.231800 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.224692    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlvnn\" (UniqueName: \"kubernetes.io/projected/13c4fdb1-f36b-4674-91b6-e84a17df72ce-kube-api-access-wlvnn\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:20.232058 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.232037    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:20.232166 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.232154    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c4fdb1-f36b-4674-91b6-e84a17df72ce-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:20.232253 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.232242    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c4fdb1-f36b-4674-91b6-e84a17df72ce-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:20.302074 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.302036    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:20.697321 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.697227    5224 generic.go:356] "Generic (PLEG): container finished" podID="13c4fdb1-f36b-4674-91b6-e84a17df72ce" containerID="e6534369605532efe0d64c46f51d2899282fd77ee5e4dd0bdc226cb4e706540d" exitCode=0
Nov 02 21:03:20.697321 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.697291    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"
Nov 02 21:03:20.697557 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.697314    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw" event={"ID":"13c4fdb1-f36b-4674-91b6-e84a17df72ce","Type":"ContainerDied","Data":"e6534369605532efe0d64c46f51d2899282fd77ee5e4dd0bdc226cb4e706540d"}
Nov 02 21:03:20.697557 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.697379    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw" event={"ID":"13c4fdb1-f36b-4674-91b6-e84a17df72ce","Type":"ContainerDied","Data":"94f23b939ccb7015bc444b1519b09c05a37ec4dc56bd1f63c5e0d2fe5152f82c"}
Nov 02 21:03:20.697557 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.697401    5224 scope.go:122] "RemoveContainer" containerID="e6534369605532efe0d64c46f51d2899282fd77ee5e4dd0bdc226cb4e706540d"
Nov 02 21:03:20.708007 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.707988    5224 scope.go:122] "RemoveContainer" containerID="e6534369605532efe0d64c46f51d2899282fd77ee5e4dd0bdc226cb4e706540d"
Nov 02 21:03:20.708303 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:03:20.708286    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6534369605532efe0d64c46f51d2899282fd77ee5e4dd0bdc226cb4e706540d\": container with ID starting with e6534369605532efe0d64c46f51d2899282fd77ee5e4dd0bdc226cb4e706540d not found: ID does not exist" containerID="e6534369605532efe0d64c46f51d2899282fd77ee5e4dd0bdc226cb4e706540d"
Nov 02 21:03:20.708349 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.708312    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6534369605532efe0d64c46f51d2899282fd77ee5e4dd0bdc226cb4e706540d"} err="failed to get container status \"e6534369605532efe0d64c46f51d2899282fd77ee5e4dd0bdc226cb4e706540d\": rpc error: code = NotFound desc = could not find container \"e6534369605532efe0d64c46f51d2899282fd77ee5e4dd0bdc226cb4e706540d\": container with ID starting with e6534369605532efe0d64c46f51d2899282fd77ee5e4dd0bdc226cb4e706540d not found: ID does not exist"
Nov 02 21:03:20.865165 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.865133    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"]
Nov 02 21:03:20.916180 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:20.916147    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-801923-55966bfd5d-6nvlw"]
Nov 02 21:03:21.167120 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:21.167087    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c4fdb1-f36b-4674-91b6-e84a17df72ce" path="/var/lib/kubelet/pods/13c4fdb1-f36b-4674-91b6-e84a17df72ce/volumes"
Nov 02 21:03:21.855464 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:21.855432    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:26.660442 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:26.660406    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:26.849499 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:26.849461    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:27.030483 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:27.030449    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:27.358394 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:03:27.358307    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:03:28.223180 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:28.223143    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:28.737359 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:28.737324    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"]
Nov 02 21:03:28.737626 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:28.737563    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775" podUID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerName="istio-proxy" containerID="cri-o://0a1a5eac6f67b6c7ad4594847bee89e0bdbb45ccd3109bb46ab7150f77d9c690" gracePeriod=30
Nov 02 21:03:28.804275 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:28.804240    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:32.211080 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:32.211044    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:33.201517 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:33.201476    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:33.998394 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:33.998363    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 21:03:34.088230 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.088190    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-token\" (UniqueName: \"kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-token\") pod \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") "
Nov 02 21:03:34.088447 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.088281    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-certs\") pod \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") "
Nov 02 21:03:34.088447 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.088331    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-credential-socket\") pod \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") "
Nov 02 21:03:34.088447 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.088364    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-data\") pod \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") "
Nov 02 21:03:34.088447 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.088392    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-socket\") pod \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") "
Nov 02 21:03:34.088447 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.088442    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-kube-api-access-59rll\" (UniqueName: \"kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-kube-api-access-59rll\") pod \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") "
Nov 02 21:03:34.088715 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.088491    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-envoy\") pod \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") "
Nov 02 21:03:34.088715 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.088548    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istiod-ca-cert\") pod \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") "
Nov 02 21:03:34.088715 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.088583    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-podinfo\") pod \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\" (UID: \"232a5dd5-ff8a-4aa4-960c-ccaf79722eda\") "
Nov 02 21:03:34.088715 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.088619    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-certs" pod "232a5dd5-ff8a-4aa4-960c-ccaf79722eda" (UID: "232a5dd5-ff8a-4aa4-960c-ccaf79722eda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:03:34.088715 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.088643    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-data" pod "232a5dd5-ff8a-4aa4-960c-ccaf79722eda" (UID: "232a5dd5-ff8a-4aa4-960c-ccaf79722eda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:03:34.088715 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.088680    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-credential-socket" pod "232a5dd5-ff8a-4aa4-960c-ccaf79722eda" (UID: "232a5dd5-ff8a-4aa4-960c-ccaf79722eda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:03:34.088715 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.088693    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-socket" pod "232a5dd5-ff8a-4aa4-960c-ccaf79722eda" (UID: "232a5dd5-ff8a-4aa4-960c-ccaf79722eda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:03:34.089104 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.089024    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istiod-ca-cert" pod "232a5dd5-ff8a-4aa4-960c-ccaf79722eda" (UID: "232a5dd5-ff8a-4aa4-960c-ccaf79722eda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:03:34.089173 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.089162    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:34.089231 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.089176    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:34.089231 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.089186    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:34.089231 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.089196    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:34.089231 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.089210    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:34.090809 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.090759    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-envoy" pod "232a5dd5-ff8a-4aa4-960c-ccaf79722eda" (UID: "232a5dd5-ff8a-4aa4-960c-ccaf79722eda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:03:34.090939 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.090806    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-podinfo" pod "232a5dd5-ff8a-4aa4-960c-ccaf79722eda" (UID: "232a5dd5-ff8a-4aa4-960c-ccaf79722eda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:03:34.091023 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.090988    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-token" pod "232a5dd5-ff8a-4aa4-960c-ccaf79722eda" (UID: "232a5dd5-ff8a-4aa4-960c-ccaf79722eda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:03:34.091079 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.091024    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-kube-api-access-59rll" pod "232a5dd5-ff8a-4aa4-960c-ccaf79722eda" (UID: "232a5dd5-ff8a-4aa4-960c-ccaf79722eda"). InnerVolumeSpecName "kube-api-access-59rll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:03:34.189890 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.189860    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:34.189890 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.189886    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59rll\" (UniqueName: \"kubernetes.io/projected/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-kube-api-access-59rll\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:34.189890 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.189898    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:34.190143 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.189908    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232a5dd5-ff8a-4aa4-960c-ccaf79722eda-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:34.782324 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.782290    5224 generic.go:356] "Generic (PLEG): container finished" podID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerID="0a1a5eac6f67b6c7ad4594847bee89e0bdbb45ccd3109bb46ab7150f77d9c690" exitCode=0
Nov 02 21:03:34.782507 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.782363    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"
Nov 02 21:03:34.782507 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.782370    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775" event={"ID":"232a5dd5-ff8a-4aa4-960c-ccaf79722eda","Type":"ContainerDied","Data":"0a1a5eac6f67b6c7ad4594847bee89e0bdbb45ccd3109bb46ab7150f77d9c690"}
Nov 02 21:03:34.782507 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.782458    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775" event={"ID":"232a5dd5-ff8a-4aa4-960c-ccaf79722eda","Type":"ContainerDied","Data":"bb7ead1f51928aaab29772bf3f9dbea33ad699d37db449ccbf91356e725bdf03"}
Nov 02 21:03:34.782507 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.782481    5224 scope.go:122] "RemoveContainer" containerID="0a1a5eac6f67b6c7ad4594847bee89e0bdbb45ccd3109bb46ab7150f77d9c690"
Nov 02 21:03:34.811717 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.811683    5224 scope.go:122] "RemoveContainer" containerID="bb94a61cdfb42855a61bc8eb50a2b45eb8f6f603cfe7810e81ee0b20431c6630"
Nov 02 21:03:34.821087 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.821068    5224 scope.go:122] "RemoveContainer" containerID="0a1a5eac6f67b6c7ad4594847bee89e0bdbb45ccd3109bb46ab7150f77d9c690"
Nov 02 21:03:34.821399 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:03:34.821382    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1a5eac6f67b6c7ad4594847bee89e0bdbb45ccd3109bb46ab7150f77d9c690\": container with ID starting with 0a1a5eac6f67b6c7ad4594847bee89e0bdbb45ccd3109bb46ab7150f77d9c690 not found: ID does not exist" containerID="0a1a5eac6f67b6c7ad4594847bee89e0bdbb45ccd3109bb46ab7150f77d9c690"
Nov 02 21:03:34.821446 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.821411    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1a5eac6f67b6c7ad4594847bee89e0bdbb45ccd3109bb46ab7150f77d9c690"} err="failed to get container status \"0a1a5eac6f67b6c7ad4594847bee89e0bdbb45ccd3109bb46ab7150f77d9c690\": rpc error: code = NotFound desc = could not find container \"0a1a5eac6f67b6c7ad4594847bee89e0bdbb45ccd3109bb46ab7150f77d9c690\": container with ID starting with 0a1a5eac6f67b6c7ad4594847bee89e0bdbb45ccd3109bb46ab7150f77d9c690 not found: ID does not exist"
Nov 02 21:03:34.821446 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.821430    5224 scope.go:122] "RemoveContainer" containerID="bb94a61cdfb42855a61bc8eb50a2b45eb8f6f603cfe7810e81ee0b20431c6630"
Nov 02 21:03:34.821720 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:03:34.821694    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb94a61cdfb42855a61bc8eb50a2b45eb8f6f603cfe7810e81ee0b20431c6630\": container with ID starting with bb94a61cdfb42855a61bc8eb50a2b45eb8f6f603cfe7810e81ee0b20431c6630 not found: ID does not exist" containerID="bb94a61cdfb42855a61bc8eb50a2b45eb8f6f603cfe7810e81ee0b20431c6630"
Nov 02 21:03:34.821780 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.821733    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb94a61cdfb42855a61bc8eb50a2b45eb8f6f603cfe7810e81ee0b20431c6630"} err="failed to get container status \"bb94a61cdfb42855a61bc8eb50a2b45eb8f6f603cfe7810e81ee0b20431c6630\": rpc error: code = NotFound desc = could not find container \"bb94a61cdfb42855a61bc8eb50a2b45eb8f6f603cfe7810e81ee0b20431c6630\": container with ID starting with bb94a61cdfb42855a61bc8eb50a2b45eb8f6f603cfe7810e81ee0b20431c6630 not found: ID does not exist"
Nov 02 21:03:34.950061 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:34.950026    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"]
Nov 02 21:03:35.002185 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:35.002150    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-801923-openshift-default-699785fb7-ts775"]
Nov 02 21:03:35.168479 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:35.168402    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" path="/var/lib/kubelet/pods/232a5dd5-ff8a-4aa4-960c-ccaf79722eda/volumes"
Nov 02 21:03:37.166063 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:37.166025    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:41.764575 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:41.764540    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:03:41.977464 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:41.977430    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:46.853093 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:46.853056    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:51.652072 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:51.651962    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:56.552021 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:03:56.551985    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:03:57.358211 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:03:57.358175    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:04:01.376717 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:01.376680    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:06.288626 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:06.288570    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:08.796687 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:08.796644    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:10.116794 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:10.116730    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:10.835422 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:10.835389    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:11.052909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:11.052875    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:15.565213 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:15.565176    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:15.941097 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:15.941012    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:20.917133 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:20.917095    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:25.729916 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:25.729885    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:27.361259 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:04:27.361224    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:04:30.535546 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:30.535510    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:33.267443 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:33.267404    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:33.697490 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:33.697392    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:33.880172 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:33.880133    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:35.467442 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:35.467409    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:39.345533 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:39.345496    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:39.527057 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:39.527024    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:40.369486 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:40.369431    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:41.667852 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:41.667813    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:45.277889 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:45.277844    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:50.170845 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:50.170805    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:50.348358 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:50.348329    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:04:53.358793 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:53.352859    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:04:53.358793 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:53.352935    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:04:55.092310 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:55.092276    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:04:57.358153 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:04:57.358112    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:04:59.905214 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:04:59.905175    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:04.705379 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:04.705337    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:09.560363 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:09.560315    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:10.510784 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:10.510736    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:11.399178 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:11.399133    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:14.400873 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:14.400834    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:19.276891 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:19.276798    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:24.149372 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:24.149333    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:27.360248 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:05:27.360204    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:05:28.995141 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:28.995101    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:30.921711 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:30.921669    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:31.110400 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:31.110360    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:33.905887 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:33.905837    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:38.853589 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:38.853546    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:39.402892 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:39.402858    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:39.983505 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:39.983464    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:40.455042 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:40.454991    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:43.804994 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:43.804948    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:44.365154 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:44.365118    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:44.550329 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:44.550291    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:44.733252 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:44.733155    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:45.420883 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:45.420840    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:49.355327 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:49.355284    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:50.947909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:50.947871    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:51.121196 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:51.121158    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:54.254746 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:54.254699    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:57.359656 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:05:57.359618    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:05:57.893145 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:57.893104    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:05:58.798884 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:58.798848    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:05:59.170343 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:05:59.170254    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:01.069722 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:01.069679    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:04.419938 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:04.419905    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:09.290710 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:09.290677    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:10.783760 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:10.783713    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:11.511127 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:11.511076    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:14.103986 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:14.103947    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:18.985610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:18.985555    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:23.856502 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:23.856463    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:27.358531 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:06:27.358497    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:06:28.734291 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:28.734247    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:32.360522 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:32.360487    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:33.572010 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:33.571970    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:38.154049 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:38.154015    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:38.562823 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:38.562755    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:43.450002 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:43.449960    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:48.196303 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:48.196218    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:50.050015 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:50.049981    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:06:53.293235 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:53.293190    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:53.479126 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:53.479078    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:53.665879 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:53.665761    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:06:57.359354 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:06:57.359315    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:06:58.462023 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:06:58.461986    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:03.367055 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:03.366988    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:04.531909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.531871    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"]
Nov 02 21:07:04.532350 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532337    5224 cpu_manager.go:414] "containerMap: removing container" podUID="dffa4399-a7ef-4f19-8177-42c954bfbc40" containerName="maas-api"
Nov 02 21:07:04.532396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532351    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffa4399-a7ef-4f19-8177-42c954bfbc40" containerName="maas-api"
Nov 02 21:07:04.532396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532362    5224 cpu_manager.go:414] "containerMap: removing container" podUID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerName="istio-proxy"
Nov 02 21:07:04.532396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532369    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerName="istio-proxy"
Nov 02 21:07:04.532396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532377    5224 cpu_manager.go:414] "containerMap: removing container" podUID="89d66f8a-6db0-4cab-9c46-d28df83c9953" containerName="cleanup"
Nov 02 21:07:04.532396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532383    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d66f8a-6db0-4cab-9c46-d28df83c9953" containerName="cleanup"
Nov 02 21:07:04.532396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532391    5224 cpu_manager.go:414] "containerMap: removing container" podUID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581" containerName="cleanup"
Nov 02 21:07:04.532396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532396    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532404    5224 cpu_manager.go:414] "containerMap: removing container" podUID="89d66f8a-6db0-4cab-9c46-d28df83c9953" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532408    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d66f8a-6db0-4cab-9c46-d28df83c9953" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532424    5224 cpu_manager.go:414] "containerMap: removing container" podUID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerName="istio-proxy"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532428    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerName="istio-proxy"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532436    5224 cpu_manager.go:414] "containerMap: removing container" podUID="13c4fdb1-f36b-4674-91b6-e84a17df72ce" containerName="maas-api"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532441    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c4fdb1-f36b-4674-91b6-e84a17df72ce" containerName="maas-api"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532451    5224 cpu_manager.go:414] "containerMap: removing container" podUID="5840d9e3-7b79-408a-ab57-2dd8ca622771" containerName="maas-api"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532456    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5840d9e3-7b79-408a-ab57-2dd8ca622771" containerName="maas-api"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532461    5224 cpu_manager.go:414] "containerMap: removing container" podUID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532466    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532472    5224 cpu_manager.go:414] "containerMap: removing container" podUID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532477    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532482    5224 cpu_manager.go:414] "containerMap: removing container" podUID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532487    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532495    5224 cpu_manager.go:414] "containerMap: removing container" podUID="72cd7435-3e8f-4e42-8c96-08f4eafa8437" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532499    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cd7435-3e8f-4e42-8c96-08f4eafa8437" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532510    5224 cpu_manager.go:414] "containerMap: removing container" podUID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532515    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532524    5224 cpu_manager.go:414] "containerMap: removing container" podUID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532529    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532536    5224 cpu_manager.go:414] "containerMap: removing container" podUID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532540    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532546    5224 cpu_manager.go:414] "containerMap: removing container" podUID="72cd7435-3e8f-4e42-8c96-08f4eafa8437" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532550    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cd7435-3e8f-4e42-8c96-08f4eafa8437" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532559    5224 cpu_manager.go:414] "containerMap: removing container" podUID="972297c3-0639-4570-8636-5d29249c866f" containerName="maas-api"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532563    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972297c3-0639-4570-8636-5d29249c866f" containerName="maas-api"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532571    5224 cpu_manager.go:414] "containerMap: removing container" podUID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532576    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532583    5224 cpu_manager.go:414] "containerMap: removing container" podUID="72cd7435-3e8f-4e42-8c96-08f4eafa8437" containerName="cleanup"
Nov 02 21:07:04.532614 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532587    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cd7435-3e8f-4e42-8c96-08f4eafa8437" containerName="cleanup"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532657    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122" containerName="cleanup"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532665    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581" containerName="cleanup"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532671    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a" containerName="cleanup"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532678    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cd7435-3e8f-4e42-8c96-08f4eafa8437" containerName="cleanup"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532684    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="972297c3-0639-4570-8636-5d29249c866f" containerName="maas-api"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532690    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122" containerName="cleanup"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532696    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581" containerName="cleanup"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532702    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffa4399-a7ef-4f19-8177-42c954bfbc40" containerName="maas-api"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532707    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerName="istio-proxy"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532712    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d66f8a-6db0-4cab-9c46-d28df83c9953" containerName="cleanup"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532716    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581" containerName="cleanup"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532722    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d66f8a-6db0-4cab-9c46-d28df83c9953" containerName="cleanup"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532729    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="232a5dd5-ff8a-4aa4-960c-ccaf79722eda" containerName="istio-proxy"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532735    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c4fdb1-f36b-4674-91b6-e84a17df72ce" containerName="maas-api"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532742    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5840d9e3-7b79-408a-ab57-2dd8ca622771" containerName="maas-api"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532748    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a" containerName="cleanup"
Nov 02 21:07:04.533586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.532754    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cd7435-3e8f-4e42-8c96-08f4eafa8437" containerName="cleanup"
Nov 02 21:07:04.535979 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.535962    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.616083 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.616046    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"]
Nov 02 21:07:04.641286 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.641253    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-44a484\"/\"kube-root-ca.crt\""
Nov 02 21:07:04.641455 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.641429    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-44a484\"/\"openshift-service-ca.crt\""
Nov 02 21:07:04.641670 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.641651    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-44a484\"/\"rate-test-model-44a484-kserve-self-signed-certs\""
Nov 02 21:07:04.641720 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.641683    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-44a484\"/\"default-dockercfg-wfkr9\""
Nov 02 21:07:04.642517 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.642500    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-model-cache\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.642566 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.642531    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-home\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.642566 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.642559    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s88mj\" (UniqueName: \"kubernetes.io/projected/de83a78c-767c-4f9f-a0b4-3560413bbba7-kube-api-access-s88mj\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.642792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.642762    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-dshm\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.642838 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.642819    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de83a78c-767c-4f9f-a0b4-3560413bbba7-tls-certs\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.642891 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.642877    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-tmp-dir\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.692626 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.692580    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:04.744110 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.744071    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-model-cache\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.744325 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.744117    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-home\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.744325 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.744154    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88mj\" (UniqueName: \"kubernetes.io/projected/de83a78c-767c-4f9f-a0b4-3560413bbba7-kube-api-access-s88mj\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.744456 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.744333    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-dshm\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.744456 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.744386    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de83a78c-767c-4f9f-a0b4-3560413bbba7-tls-certs\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.744456 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.744421    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-tmp-dir\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.744638 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.744550    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-model-cache\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.744638 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.744606    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-home\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.744750 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.744734    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-tmp-dir\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.746545 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.746522    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-dshm\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.746843 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.746827    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de83a78c-767c-4f9f-a0b4-3560413bbba7-tls-certs\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.802487 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.802411    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88mj\" (UniqueName: \"kubernetes.io/projected/de83a78c-767c-4f9f-a0b4-3560413bbba7-kube-api-access-s88mj\") pod \"rate-test-model-44a484-kserve-55fb866d97-m4hm4\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") " pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:04.848024 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:04.847986    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:05.222795 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:07:05.222735    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde83a78c_767c_4f9f_a0b4_3560413bbba7.slice/crio-a12f5ba0bd8e1f7b58f46aba83e27bfdb90960e9262fc81fa06b4ae191a654d7 WatchSource:0}: Error finding container a12f5ba0bd8e1f7b58f46aba83e27bfdb90960e9262fc81fa06b4ae191a654d7: Status 404 returned error can't find the container with id a12f5ba0bd8e1f7b58f46aba83e27bfdb90960e9262fc81fa06b4ae191a654d7
Nov 02 21:07:05.226361 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:05.226327    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"]
Nov 02 21:07:06.094461 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:06.094423    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4" event={"ID":"de83a78c-767c-4f9f-a0b4-3560413bbba7","Type":"ContainerStarted","Data":"71ced0fcda7a7cc896e13706091e2b55d815a95e4e559ecfed8a3b444b3f8dad"}
Nov 02 21:07:06.094863 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:06.094476    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4" event={"ID":"de83a78c-767c-4f9f-a0b4-3560413bbba7","Type":"ContainerStarted","Data":"a12f5ba0bd8e1f7b58f46aba83e27bfdb90960e9262fc81fa06b4ae191a654d7"}
Nov 02 21:07:06.211443 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:06.211390    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4" podStartSLOduration=2.211374898 podStartE2EDuration="2.211374898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:07:06.210087804 +0000 UTC m=+4707.207776414" watchObservedRunningTime="2026-11-02 21:07:06.211374898 +0000 UTC m=+4707.209063507"
Nov 02 21:07:06.848298 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:06.848256    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:08.170891 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:08.170840    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:11.116894 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:11.116834    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:11.849297 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:11.849266    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:13.070884 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:13.070847    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:14.848613 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:14.848583    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:14.860892 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:14.860864    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:15.157758 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:15.157672    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:07:17.968083 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:17.968045    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:22.972587 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:22.972549    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:27.358321 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:07:27.358282    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:07:27.864419 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:27.864354    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:32.829100 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:32.829053    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:37.661089 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:37.661048    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:42.466969 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:42.466905    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:43.833109 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:43.833072    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:44.006827 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:44.006794    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:45.762784 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:45.762720    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:45.936364 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:45.936324    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:47.304056 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:47.304005    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:48.771489 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:48.771454    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:07:52.178793 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.177287    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:52.863287 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.863252    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"]
Nov 02 21:07:52.863751 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.863737    5224 cpu_manager.go:414] "containerMap: removing container" podUID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581" containerName="cleanup"
Nov 02 21:07:52.863821 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.863752    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3ebb63-cca2-47c5-be89-1dbfdd34c581" containerName="cleanup"
Nov 02 21:07:52.863821 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.863777    5224 cpu_manager.go:414] "containerMap: removing container" podUID="89d66f8a-6db0-4cab-9c46-d28df83c9953" containerName="cleanup"
Nov 02 21:07:52.863821 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.863784    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d66f8a-6db0-4cab-9c46-d28df83c9953" containerName="cleanup"
Nov 02 21:07:52.863940 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.863874    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d66f8a-6db0-4cab-9c46-d28df83c9953" containerName="cleanup"
Nov 02 21:07:52.863940 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.863887    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4383ae-73a5-45a1-979d-16a9dbb7e83a" containerName="cleanup"
Nov 02 21:07:52.863940 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.863896    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cd7435-3e8f-4e42-8c96-08f4eafa8437" containerName="cleanup"
Nov 02 21:07:52.863940 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.863902    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d2d907-945a-4fd8-bb27-5a4c78b2b122" containerName="cleanup"
Nov 02 21:07:52.866920 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.866904    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:52.925078 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.925049    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"]
Nov 02 21:07:52.973717 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.973676    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-09ecd0\"/\"openshift-service-ca.crt\""
Nov 02 21:07:52.973950 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.973831    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-09ecd0\"/\"kube-root-ca.crt\""
Nov 02 21:07:52.973950 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.973854    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-09ecd0\"/\"rate-test-model-09ecd0-kserve-self-signed-certs\""
Nov 02 21:07:52.973950 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.973893    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-09ecd0\"/\"default-dockercfg-l9gt8\""
Nov 02 21:07:52.988481 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.988454    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-model-cache\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:52.988573 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.988494    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpbw6\" (UniqueName: \"kubernetes.io/projected/bc611f03-fc41-41c6-a631-d0b344a7c8f9-kube-api-access-zpbw6\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:52.988573 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.988518    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tmp-dir\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:52.988703 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.988682    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-dshm\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:52.988763 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.988738    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tls-certs\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:52.988942 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:52.988927    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-home\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.066988 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.066954    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:53.090245 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.090207    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-dshm\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.090444 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.090252    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tls-certs\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.090444 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.090350    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-home\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.090444 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.090384    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-model-cache\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.090444 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.090400    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbw6\" (UniqueName: \"kubernetes.io/projected/bc611f03-fc41-41c6-a631-d0b344a7c8f9-kube-api-access-zpbw6\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.090444 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.090430    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tmp-dir\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.090865 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.090844    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tmp-dir\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.091113 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.091058    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-model-cache\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.091113 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.091083    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-home\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.092942 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.092922    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-dshm\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.093321 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.093300    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tls-certs\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.149254 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.149163    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbw6\" (UniqueName: \"kubernetes.io/projected/bc611f03-fc41-41c6-a631-d0b344a7c8f9-kube-api-access-zpbw6\") pod \"rate-test-model-09ecd0-kserve-cb65595c-55zzg\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") " pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.178398 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.178361    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:53.551460 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:07:53.551424    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc611f03_fc41_41c6_a631_d0b344a7c8f9.slice/crio-eeef76102c0b414b20572397d70f8fe7f77f0bfd53fe16e477f5fd8a2a975073 WatchSource:0}: Error finding container eeef76102c0b414b20572397d70f8fe7f77f0bfd53fe16e477f5fd8a2a975073: Status 404 returned error can't find the container with id eeef76102c0b414b20572397d70f8fe7f77f0bfd53fe16e477f5fd8a2a975073
Nov 02 21:07:53.554726 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:53.554704    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"]
Nov 02 21:07:54.375442 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:54.375394    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg" event={"ID":"bc611f03-fc41-41c6-a631-d0b344a7c8f9","Type":"ContainerStarted","Data":"9e41f48440d947f2f0513fa7702fd8e8180a0f31f59e5f55bc6f26b9176cc359"}
Nov 02 21:07:54.375649 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:54.375452    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg" event={"ID":"bc611f03-fc41-41c6-a631-d0b344a7c8f9","Type":"ContainerStarted","Data":"eeef76102c0b414b20572397d70f8fe7f77f0bfd53fe16e477f5fd8a2a975073"}
Nov 02 21:07:54.491745 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:54.491694    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg" podStartSLOduration=2.491678732 podStartE2EDuration="2.491678732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:07:54.490463889 +0000 UTC m=+4755.488152502" watchObservedRunningTime="2026-11-02 21:07:54.491678732 +0000 UTC m=+4755.489367382"
Nov 02 21:07:55.179234 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:55.179206    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:07:57.103019 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:07:57.102987    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:07:57.359734 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:07:57.359647    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:08:01.963172 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:01.963120    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:02.850129 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:02.850097    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:03.112201 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:03.112092    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:03.179282 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:03.179252    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:08:03.191853 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:03.191827    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:08:03.350758 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:03.350726    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:03.444132 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:03.444046    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:08:06.858630 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:06.858589    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:08.799229 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:08.799192    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:09.022505 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:09.022470    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:09.233699 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:09.233600    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:11.456201 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:11.456162    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:11.845393 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:11.845352    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:12.424903 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:12.424874    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:15.100457 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:15.100420    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:15.995701 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:15.995657    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:16.169543 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:16.169506    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:17.216250 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:17.216210    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:17.473871 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:17.473712    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:18.377606 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:18.377512    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:18.568037 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:18.567997    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:18.941294 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:18.941260    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:20.493083 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:20.493050    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:22.163131 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:22.163100    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:24.669517 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:24.669477    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:27.059779 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:27.059719    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:27.357928 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:08:27.357843    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:08:29.561258 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:29.561222    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:31.225205 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:31.225167    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:08:31.899456 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:31.899408    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:36.132617 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:36.132579    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:37.975794 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:37.975734    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:42.781232 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:42.781183    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:43.183607 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:43.183515    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:43.406345 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:43.406279    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:44.718127 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:44.718076    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:45.260409 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:45.260376    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"]
Nov 02 21:08:45.260637 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:45.260600    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh" podUID="4b4880ac-7310-4d51-86f2-75634f425b28" containerName="istio-proxy" containerID="cri-o://5b84ea2125a171259fa0a6b7655de97f40b9e7adddb43d23c0b1801260dd6630" gracePeriod=30
Nov 02 21:08:45.320570 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:45.320534    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:46.046483 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:46.046439    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:47.957361 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:47.957325    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:50.531260 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.531201    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 21:08:50.648473 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.648438    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-socket\") pod \"4b4880ac-7310-4d51-86f2-75634f425b28\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") "
Nov 02 21:08:50.648662 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.648545    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-kube-api-access-mlm64\" (UniqueName: \"kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-kube-api-access-mlm64\") pod \"4b4880ac-7310-4d51-86f2-75634f425b28\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") "
Nov 02 21:08:50.648662 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.648578    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-credential-socket\") pod \"4b4880ac-7310-4d51-86f2-75634f425b28\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") "
Nov 02 21:08:50.648662 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.648614    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-istio-token\" (UniqueName: \"kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-istio-token\") pod \"4b4880ac-7310-4d51-86f2-75634f425b28\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") "
Nov 02 21:08:50.648950 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.648719    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-socket" pod "4b4880ac-7310-4d51-86f2-75634f425b28" (UID: "4b4880ac-7310-4d51-86f2-75634f425b28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:08:50.648950 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.648738    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4880ac-7310-4d51-86f2-75634f425b28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4880ac-7310-4d51-86f2-75634f425b28-istiod-ca-cert\") pod \"4b4880ac-7310-4d51-86f2-75634f425b28\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") "
Nov 02 21:08:50.648950 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.648803    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-envoy\") pod \"4b4880ac-7310-4d51-86f2-75634f425b28\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") "
Nov 02 21:08:50.648950 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.648883    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-data\") pod \"4b4880ac-7310-4d51-86f2-75634f425b28\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") "
Nov 02 21:08:50.648950 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.648914    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b4880ac-7310-4d51-86f2-75634f425b28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4880ac-7310-4d51-86f2-75634f425b28-istio-podinfo\") pod \"4b4880ac-7310-4d51-86f2-75634f425b28\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") "
Nov 02 21:08:50.648950 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.648948    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-certs\") pod \"4b4880ac-7310-4d51-86f2-75634f425b28\" (UID: \"4b4880ac-7310-4d51-86f2-75634f425b28\") "
Nov 02 21:08:50.649277 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.648955    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-credential-socket" pod "4b4880ac-7310-4d51-86f2-75634f425b28" (UID: "4b4880ac-7310-4d51-86f2-75634f425b28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:08:50.649277 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.649148    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-data" pod "4b4880ac-7310-4d51-86f2-75634f425b28" (UID: "4b4880ac-7310-4d51-86f2-75634f425b28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:08:50.649277 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.649170    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4880ac-7310-4d51-86f2-75634f425b28-istiod-ca-cert" pod "4b4880ac-7310-4d51-86f2-75634f425b28" (UID: "4b4880ac-7310-4d51-86f2-75634f425b28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:08:50.649450 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.649345    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-certs" pod "4b4880ac-7310-4d51-86f2-75634f425b28" (UID: "4b4880ac-7310-4d51-86f2-75634f425b28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:08:50.649552 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.649533    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:08:50.649620 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.649556    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:08:50.649620 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.649568    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4880ac-7310-4d51-86f2-75634f425b28-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:08:50.649620 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.649581    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:08:50.649620 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.649592    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:08:50.651017 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.650989    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b4880ac-7310-4d51-86f2-75634f425b28-istio-podinfo" pod "4b4880ac-7310-4d51-86f2-75634f425b28" (UID: "4b4880ac-7310-4d51-86f2-75634f425b28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:08:50.651125 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.651059    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-kube-api-access-mlm64" pod "4b4880ac-7310-4d51-86f2-75634f425b28" (UID: "4b4880ac-7310-4d51-86f2-75634f425b28"). InnerVolumeSpecName "kube-api-access-mlm64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:08:50.651266 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.651246    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-istio-token" pod "4b4880ac-7310-4d51-86f2-75634f425b28" (UID: "4b4880ac-7310-4d51-86f2-75634f425b28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:08:50.651315 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.651250    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-envoy" pod "4b4880ac-7310-4d51-86f2-75634f425b28" (UID: "4b4880ac-7310-4d51-86f2-75634f425b28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:08:50.712739 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.712702    5224 generic.go:356] "Generic (PLEG): container finished" podID="4b4880ac-7310-4d51-86f2-75634f425b28" containerID="5b84ea2125a171259fa0a6b7655de97f40b9e7adddb43d23c0b1801260dd6630" exitCode=0
Nov 02 21:08:50.712933 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.712753    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"
Nov 02 21:08:50.712933 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.712785    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh" event={"ID":"4b4880ac-7310-4d51-86f2-75634f425b28","Type":"ContainerDied","Data":"5b84ea2125a171259fa0a6b7655de97f40b9e7adddb43d23c0b1801260dd6630"}
Nov 02 21:08:50.712933 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.712857    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh" event={"ID":"4b4880ac-7310-4d51-86f2-75634f425b28","Type":"ContainerDied","Data":"a0f2a65597c8ba5e53282550a5f31618ae8a1818cf18aa1e48ec43c019148b2b"}
Nov 02 21:08:50.712933 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.712894    5224 scope.go:122] "RemoveContainer" containerID="5b84ea2125a171259fa0a6b7655de97f40b9e7adddb43d23c0b1801260dd6630"
Nov 02 21:08:50.724115 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.724097    5224 scope.go:122] "RemoveContainer" containerID="ae092d241cf7cfcbe00d35b7886d7400707e30c0f02e2202cb31ae47ad91076e"
Nov 02 21:08:50.734701 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.734682    5224 scope.go:122] "RemoveContainer" containerID="5b84ea2125a171259fa0a6b7655de97f40b9e7adddb43d23c0b1801260dd6630"
Nov 02 21:08:50.734994 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:08:50.734977    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b84ea2125a171259fa0a6b7655de97f40b9e7adddb43d23c0b1801260dd6630\": container with ID starting with 5b84ea2125a171259fa0a6b7655de97f40b9e7adddb43d23c0b1801260dd6630 not found: ID does not exist" containerID="5b84ea2125a171259fa0a6b7655de97f40b9e7adddb43d23c0b1801260dd6630"
Nov 02 21:08:50.735061 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.735003    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b84ea2125a171259fa0a6b7655de97f40b9e7adddb43d23c0b1801260dd6630"} err="failed to get container status \"5b84ea2125a171259fa0a6b7655de97f40b9e7adddb43d23c0b1801260dd6630\": rpc error: code = NotFound desc = could not find container \"5b84ea2125a171259fa0a6b7655de97f40b9e7adddb43d23c0b1801260dd6630\": container with ID starting with 5b84ea2125a171259fa0a6b7655de97f40b9e7adddb43d23c0b1801260dd6630 not found: ID does not exist"
Nov 02 21:08:50.735061 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.735023    5224 scope.go:122] "RemoveContainer" containerID="ae092d241cf7cfcbe00d35b7886d7400707e30c0f02e2202cb31ae47ad91076e"
Nov 02 21:08:50.735293 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:08:50.735278    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae092d241cf7cfcbe00d35b7886d7400707e30c0f02e2202cb31ae47ad91076e\": container with ID starting with ae092d241cf7cfcbe00d35b7886d7400707e30c0f02e2202cb31ae47ad91076e not found: ID does not exist" containerID="ae092d241cf7cfcbe00d35b7886d7400707e30c0f02e2202cb31ae47ad91076e"
Nov 02 21:08:50.735343 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.735295    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae092d241cf7cfcbe00d35b7886d7400707e30c0f02e2202cb31ae47ad91076e"} err="failed to get container status \"ae092d241cf7cfcbe00d35b7886d7400707e30c0f02e2202cb31ae47ad91076e\": rpc error: code = NotFound desc = could not find container \"ae092d241cf7cfcbe00d35b7886d7400707e30c0f02e2202cb31ae47ad91076e\": container with ID starting with ae092d241cf7cfcbe00d35b7886d7400707e30c0f02e2202cb31ae47ad91076e not found: ID does not exist"
Nov 02 21:08:50.750731 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.750708    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlm64\" (UniqueName: \"kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-kube-api-access-mlm64\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:08:50.750841 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.750732    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4880ac-7310-4d51-86f2-75634f425b28-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:08:50.750841 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.750743    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4880ac-7310-4d51-86f2-75634f425b28-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:08:50.750841 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.750751    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4880ac-7310-4d51-86f2-75634f425b28-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:08:50.880720 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.880644    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"]
Nov 02 21:08:50.882961 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:50.882941    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2c6ce9-openshift-default-85799b7dff-qkmjh"]
Nov 02 21:08:51.176210 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:51.176131    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4880ac-7310-4d51-86f2-75634f425b28" path="/var/lib/kubelet/pods/4b4880ac-7310-4d51-86f2-75634f425b28/volumes"
Nov 02 21:08:51.849600 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:51.849563    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:52.869149 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:52.869115    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:53.068631 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:53.068599    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:57.359594 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:08:57.359558    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:08:57.441630 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:57.441593    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:57.655023 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:57.654949    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:58.061464 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:08:58.061429    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:02.951922 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:02.951882    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:07.908327 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:07.908286    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:11.803948 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:11.803912    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:12.525871 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:12.525833    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:12.757754 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:12.757717    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:17.601788 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:17.601738    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:22.437031 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:22.436993    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:26.013344 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:26.013311    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:09:27.257731 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:27.257691    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:27.357443 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:09:27.357406    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:09:32.183926 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:32.183884    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:36.881824 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:36.881790    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:41.701060 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:41.701019    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:46.561553 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:46.561508    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:49.768449 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:49.768361    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:51.513683 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:51.513649    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:55.585473 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:55.585437    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"]
Nov 02 21:09:55.586017 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:55.585729    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl" podUID="375bdcc1-e847-4275-85b8-6a0197b5cb66" containerName="maas-api" containerID="cri-o://2f3e3a836d08704d91d34ec451c12f91cc766dc5fd0c2248cd0a520c3ccb4ea9" gracePeriod=30
Nov 02 21:09:55.790863 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:55.790780    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:55.866934 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:55.866900    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 21:09:55.970450 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:55.970413    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-maas-api-tls\") pod \"375bdcc1-e847-4275-85b8-6a0197b5cb66\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") "
Nov 02 21:09:55.970669 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:55.970470    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375bdcc1-e847-4275-85b8-6a0197b5cb66-kube-api-access-6bmw5\" (UniqueName: \"kubernetes.io/projected/375bdcc1-e847-4275-85b8-6a0197b5cb66-kube-api-access-6bmw5\") pod \"375bdcc1-e847-4275-85b8-6a0197b5cb66\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") "
Nov 02 21:09:55.970669 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:55.970563    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-metrics-certs\") pod \"375bdcc1-e847-4275-85b8-6a0197b5cb66\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") "
Nov 02 21:09:55.970669 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:55.970624    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375bdcc1-e847-4275-85b8-6a0197b5cb66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375bdcc1-e847-4275-85b8-6a0197b5cb66-openshift-service-ca\") pod \"375bdcc1-e847-4275-85b8-6a0197b5cb66\" (UID: \"375bdcc1-e847-4275-85b8-6a0197b5cb66\") "
Nov 02 21:09:55.971895 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:55.971865    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375bdcc1-e847-4275-85b8-6a0197b5cb66-openshift-service-ca" pod "375bdcc1-e847-4275-85b8-6a0197b5cb66" (UID: "375bdcc1-e847-4275-85b8-6a0197b5cb66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:09:55.975088 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:55.975057    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375bdcc1-e847-4275-85b8-6a0197b5cb66-kube-api-access-6bmw5" pod "375bdcc1-e847-4275-85b8-6a0197b5cb66" (UID: "375bdcc1-e847-4275-85b8-6a0197b5cb66"). InnerVolumeSpecName "kube-api-access-6bmw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:09:55.998215 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:55.997051    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-maas-api-tls" pod "375bdcc1-e847-4275-85b8-6a0197b5cb66" (UID: "375bdcc1-e847-4275-85b8-6a0197b5cb66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:09:55.999154 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:55.998564    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-metrics-certs" pod "375bdcc1-e847-4275-85b8-6a0197b5cb66" (UID: "375bdcc1-e847-4275-85b8-6a0197b5cb66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:09:56.072142 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.072108    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:56.072142 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.072136    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bmw5\" (UniqueName: \"kubernetes.io/projected/375bdcc1-e847-4275-85b8-6a0197b5cb66-kube-api-access-6bmw5\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:56.072347 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.072152    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375bdcc1-e847-4275-85b8-6a0197b5cb66-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:56.072347 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.072165    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375bdcc1-e847-4275-85b8-6a0197b5cb66-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:56.075303 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.075271    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:56.081956 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.081934    5224 generic.go:356] "Generic (PLEG): container finished" podID="375bdcc1-e847-4275-85b8-6a0197b5cb66" containerID="2f3e3a836d08704d91d34ec451c12f91cc766dc5fd0c2248cd0a520c3ccb4ea9" exitCode=0
Nov 02 21:09:56.082073 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.081992    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"
Nov 02 21:09:56.082073 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.082021    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl" event={"ID":"375bdcc1-e847-4275-85b8-6a0197b5cb66","Type":"ContainerDied","Data":"2f3e3a836d08704d91d34ec451c12f91cc766dc5fd0c2248cd0a520c3ccb4ea9"}
Nov 02 21:09:56.082163 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.082080    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl" event={"ID":"375bdcc1-e847-4275-85b8-6a0197b5cb66","Type":"ContainerDied","Data":"444f7fa31db53538f4d9d19ffa0834a0a981d599428fbd1e5e514acb62fabd6c"}
Nov 02 21:09:56.082163 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.082106    5224 scope.go:122] "RemoveContainer" containerID="2f3e3a836d08704d91d34ec451c12f91cc766dc5fd0c2248cd0a520c3ccb4ea9"
Nov 02 21:09:56.093663 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.093646    5224 scope.go:122] "RemoveContainer" containerID="2f3e3a836d08704d91d34ec451c12f91cc766dc5fd0c2248cd0a520c3ccb4ea9"
Nov 02 21:09:56.093953 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:09:56.093936    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3e3a836d08704d91d34ec451c12f91cc766dc5fd0c2248cd0a520c3ccb4ea9\": container with ID starting with 2f3e3a836d08704d91d34ec451c12f91cc766dc5fd0c2248cd0a520c3ccb4ea9 not found: ID does not exist" containerID="2f3e3a836d08704d91d34ec451c12f91cc766dc5fd0c2248cd0a520c3ccb4ea9"
Nov 02 21:09:56.094017 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.093962    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3e3a836d08704d91d34ec451c12f91cc766dc5fd0c2248cd0a520c3ccb4ea9"} err="failed to get container status \"2f3e3a836d08704d91d34ec451c12f91cc766dc5fd0c2248cd0a520c3ccb4ea9\": rpc error: code = NotFound desc = could not find container \"2f3e3a836d08704d91d34ec451c12f91cc766dc5fd0c2248cd0a520c3ccb4ea9\": container with ID starting with 2f3e3a836d08704d91d34ec451c12f91cc766dc5fd0c2248cd0a520c3ccb4ea9 not found: ID does not exist"
Nov 02 21:09:56.250359 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.250322    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"]
Nov 02 21:09:56.254168 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:56.254130    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-78bef7-f988f5d5d-f5pzl"]
Nov 02 21:09:57.173293 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:57.173257    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="375bdcc1-e847-4275-85b8-6a0197b5cb66" path="/var/lib/kubelet/pods/375bdcc1-e847-4275-85b8-6a0197b5cb66/volumes"
Nov 02 21:09:57.347227 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:57.347188    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:57.382881 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:09:57.382809    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:09:57.536441 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:57.536408    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:09:57.745110 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:09:57.745074    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:01.598913 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:01.598881    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:02.861411 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:02.861361    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:03.338226 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:03.338186    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:03.573603 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:03.573564    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:04.902394 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:04.902357    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:05.390346 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.390316    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:05.855270 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.855238    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7522-9a5907"]
Nov 02 21:10:05.855712 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.855699    5224 cpu_manager.go:414] "containerMap: removing container" podUID="375bdcc1-e847-4275-85b8-6a0197b5cb66" containerName="maas-api"
Nov 02 21:10:05.855763 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.855713    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375bdcc1-e847-4275-85b8-6a0197b5cb66" containerName="maas-api"
Nov 02 21:10:05.855763 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.855726    5224 cpu_manager.go:414] "containerMap: removing container" podUID="4b4880ac-7310-4d51-86f2-75634f425b28" containerName="istio-proxy"
Nov 02 21:10:05.855763 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.855731    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4880ac-7310-4d51-86f2-75634f425b28" containerName="istio-proxy"
Nov 02 21:10:05.855763 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.855749    5224 cpu_manager.go:414] "containerMap: removing container" podUID="4b4880ac-7310-4d51-86f2-75634f425b28" containerName="istio-proxy"
Nov 02 21:10:05.855763 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.855754    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4880ac-7310-4d51-86f2-75634f425b28" containerName="istio-proxy"
Nov 02 21:10:05.855952 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.855842    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="375bdcc1-e847-4275-85b8-6a0197b5cb66" containerName="maas-api"
Nov 02 21:10:05.855952 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.855852    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4880ac-7310-4d51-86f2-75634f425b28" containerName="istio-proxy"
Nov 02 21:10:05.855952 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.855861    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4880ac-7310-4d51-86f2-75634f425b28" containerName="istio-proxy"
Nov 02 21:10:05.857489 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.857467    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:05.859257 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.859241    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-9a5907"
Nov 02 21:10:05.913749 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.913721    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7522-9a5907"]
Nov 02 21:10:05.962162 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.962139    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bkzxn\""
Nov 02 21:10:05.979622 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:05.979594    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj8r7\" (UniqueName: \"kubernetes.io/projected/8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43-kube-api-access-dj8r7\") pod \"test-curl-7522-9a5907\" (UID: \"8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43\") " pod="opendatahub/test-curl-7522-9a5907"
Nov 02 21:10:06.080201 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:06.080167    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8r7\" (UniqueName: \"kubernetes.io/projected/8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43-kube-api-access-dj8r7\") pod \"test-curl-7522-9a5907\" (UID: \"8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43\") " pod="opendatahub/test-curl-7522-9a5907"
Nov 02 21:10:06.137751 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:06.137682    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj8r7\" (UniqueName: \"kubernetes.io/projected/8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43-kube-api-access-dj8r7\") pod \"test-curl-7522-9a5907\" (UID: \"8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43\") " pod="opendatahub/test-curl-7522-9a5907"
Nov 02 21:10:06.170691 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:06.170664    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-9a5907"
Nov 02 21:10:06.536732 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:10:06.536697    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a691c16_0a1b_43d7_a2d6_6b0d5aa22f43.slice/crio-b0ee3b81ff957ec4529b36a74d7318520d08c1fb1511fd9751e9823ee8799353 WatchSource:0}: Error finding container b0ee3b81ff957ec4529b36a74d7318520d08c1fb1511fd9751e9823ee8799353: Status 404 returned error can't find the container with id b0ee3b81ff957ec4529b36a74d7318520d08c1fb1511fd9751e9823ee8799353
Nov 02 21:10:06.540287 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:06.540257    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7522-9a5907"]
Nov 02 21:10:07.148223 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:07.148183    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7522-9a5907" event={"ID":"8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43","Type":"ContainerStarted","Data":"710ff59de629dc4435fdf349bc0973352f3d2556c103617081efa966d391a3f4"}
Nov 02 21:10:07.148223 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:07.148230    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7522-9a5907" event={"ID":"8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43","Type":"ContainerStarted","Data":"b0ee3b81ff957ec4529b36a74d7318520d08c1fb1511fd9751e9823ee8799353"}
Nov 02 21:10:07.262150 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:07.262102    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7522-9a5907" podStartSLOduration=2.262087622 podStartE2EDuration="2.262087622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:10:07.260705406 +0000 UTC m=+4888.258394015" watchObservedRunningTime="2026-11-02 21:10:07.262087622 +0000 UTC m=+4888.259776230"
Nov 02 21:10:08.288976 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:08.288925    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:08.699159 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:08.699081    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7522-9a5907"]
Nov 02 21:10:08.703421 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:08.703388    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7522-9a5907"]
Nov 02 21:10:09.159497 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:09.159443    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7522-9a5907" podUID="8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43" containerName="test-curl-7522-9a5907" containerID="cri-o://710ff59de629dc4435fdf349bc0973352f3d2556c103617081efa966d391a3f4" gracePeriod=2
Nov 02 21:10:09.211963 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:09.211918    5224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7522-9a5907\" is forbidden: User \"system:node:ip-10-0-134-196.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-196.us-west-2.compute.internal' and this object" podUID="8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43" pod="opendatahub/test-curl-7522-9a5907"
Nov 02 21:10:10.779151 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:10.779110    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"]
Nov 02 21:10:10.954288 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:10.954249    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:11.408204 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:11.408175    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-9a5907"
Nov 02 21:10:11.460597 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:11.460524    5224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7522-9a5907\" is forbidden: User \"system:node:ip-10-0-134-196.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-196.us-west-2.compute.internal' and this object" podUID="8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43" pod="opendatahub/test-curl-7522-9a5907"
Nov 02 21:10:11.544749 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:11.544714    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43-kube-api-access-dj8r7\" (UniqueName: \"kubernetes.io/projected/8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43-kube-api-access-dj8r7\") pod \"8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43\" (UID: \"8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43\") "
Nov 02 21:10:11.546832 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:11.546805    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43-kube-api-access-dj8r7" pod "8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43" (UID: "8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43"). InnerVolumeSpecName "kube-api-access-dj8r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:10:11.646554 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:11.646515    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj8r7\" (UniqueName: \"kubernetes.io/projected/8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43-kube-api-access-dj8r7\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:12.141352 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:12.141320    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:12.188873 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:12.188837    5224 generic.go:356] "Generic (PLEG): container finished" podID="8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43" containerID="710ff59de629dc4435fdf349bc0973352f3d2556c103617081efa966d391a3f4" exitCode=137
Nov 02 21:10:12.189069 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:12.188889    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-9a5907"
Nov 02 21:10:12.189069 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:12.188937    5224 scope.go:122] "RemoveContainer" containerID="710ff59de629dc4435fdf349bc0973352f3d2556c103617081efa966d391a3f4"
Nov 02 21:10:12.199694 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:12.199670    5224 scope.go:122] "RemoveContainer" containerID="710ff59de629dc4435fdf349bc0973352f3d2556c103617081efa966d391a3f4"
Nov 02 21:10:12.200048 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:12.200021    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"710ff59de629dc4435fdf349bc0973352f3d2556c103617081efa966d391a3f4\": container with ID starting with 710ff59de629dc4435fdf349bc0973352f3d2556c103617081efa966d391a3f4 not found: ID does not exist" containerID="710ff59de629dc4435fdf349bc0973352f3d2556c103617081efa966d391a3f4"
Nov 02 21:10:12.200119 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:12.200048    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"710ff59de629dc4435fdf349bc0973352f3d2556c103617081efa966d391a3f4"} err="failed to get container status \"710ff59de629dc4435fdf349bc0973352f3d2556c103617081efa966d391a3f4\": rpc error: code = NotFound desc = could not find container \"710ff59de629dc4435fdf349bc0973352f3d2556c103617081efa966d391a3f4\": container with ID starting with 710ff59de629dc4435fdf349bc0973352f3d2556c103617081efa966d391a3f4 not found: ID does not exist"
Nov 02 21:10:12.241488 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:12.241456    5224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7522-9a5907\" is forbidden: User \"system:node:ip-10-0-134-196.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-196.us-west-2.compute.internal' and this object" podUID="8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43" pod="opendatahub/test-curl-7522-9a5907"
Nov 02 21:10:12.293806 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:12.293750    5224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7522-9a5907\" is forbidden: User \"system:node:ip-10-0-134-196.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-196.us-west-2.compute.internal' and this object" podUID="8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43" pod="opendatahub/test-curl-7522-9a5907"
Nov 02 21:10:12.893619 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:12.893576    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:13.074263 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:13.074214    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:13.167389 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:13.167305    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43" path="/var/lib/kubelet/pods/8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43/volumes"
Nov 02 21:10:13.276703 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:13.276667    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:18.157445 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:18.157404    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:18.896131 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:18.896096    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:19.729525 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:19.729485    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:19.909734 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:19.909694    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:20.293212 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:20.293174    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:10:22.819314 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:22.819274    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:25.003786 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:25.003731    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:25.177892 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:25.177843    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:25.804862 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:25.804816    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6" podUID="f67432d1-22af-458b-bd57-002a02c62a5d" containerName="main" containerID="cri-o://1d8bddd2e248539e1af43882fb53681d9dd509f9570ec2e3cef6b6b0f42586dd" gracePeriod=45
Nov 02 21:10:26.071332 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.071270    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 21:10:26.225466 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.225431    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-home\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-home\") pod \"f67432d1-22af-458b-bd57-002a02c62a5d\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") "
Nov 02 21:10:26.225466 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.225472    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-model-cache\") pod \"f67432d1-22af-458b-bd57-002a02c62a5d\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") "
Nov 02 21:10:26.225743 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.225521    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67432d1-22af-458b-bd57-002a02c62a5d-kube-api-access-9995f\" (UniqueName: \"kubernetes.io/projected/f67432d1-22af-458b-bd57-002a02c62a5d-kube-api-access-9995f\") pod \"f67432d1-22af-458b-bd57-002a02c62a5d\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") "
Nov 02 21:10:26.225743 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.225607    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-dshm\") pod \"f67432d1-22af-458b-bd57-002a02c62a5d\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") "
Nov 02 21:10:26.225743 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.225670    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67432d1-22af-458b-bd57-002a02c62a5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f67432d1-22af-458b-bd57-002a02c62a5d-tls-certs\") pod \"f67432d1-22af-458b-bd57-002a02c62a5d\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") "
Nov 02 21:10:26.225743 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.225728    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-tmp-dir\") pod \"f67432d1-22af-458b-bd57-002a02c62a5d\" (UID: \"f67432d1-22af-458b-bd57-002a02c62a5d\") "
Nov 02 21:10:26.226007 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.225741    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-home" pod "f67432d1-22af-458b-bd57-002a02c62a5d" (UID: "f67432d1-22af-458b-bd57-002a02c62a5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:26.226007 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.225848    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-model-cache" pod "f67432d1-22af-458b-bd57-002a02c62a5d" (UID: "f67432d1-22af-458b-bd57-002a02c62a5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:26.226125 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.226092    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-tmp-dir" pod "f67432d1-22af-458b-bd57-002a02c62a5d" (UID: "f67432d1-22af-458b-bd57-002a02c62a5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:26.226260 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.226236    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:26.226260 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.226259    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:26.226458 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.226274    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:26.228231 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.228205    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67432d1-22af-458b-bd57-002a02c62a5d-kube-api-access-9995f" pod "f67432d1-22af-458b-bd57-002a02c62a5d" (UID: "f67432d1-22af-458b-bd57-002a02c62a5d"). InnerVolumeSpecName "kube-api-access-9995f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:10:26.228359 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.228333    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67432d1-22af-458b-bd57-002a02c62a5d-tls-certs" pod "f67432d1-22af-458b-bd57-002a02c62a5d" (UID: "f67432d1-22af-458b-bd57-002a02c62a5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:10:26.228423 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.228354    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-dshm" pod "f67432d1-22af-458b-bd57-002a02c62a5d" (UID: "f67432d1-22af-458b-bd57-002a02c62a5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:26.279189 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.279146    5224 generic.go:356] "Generic (PLEG): container finished" podID="f67432d1-22af-458b-bd57-002a02c62a5d" containerID="1d8bddd2e248539e1af43882fb53681d9dd509f9570ec2e3cef6b6b0f42586dd" exitCode=0
Nov 02 21:10:26.279400 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.279198    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"
Nov 02 21:10:26.279400 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.279234    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6" event={"ID":"f67432d1-22af-458b-bd57-002a02c62a5d","Type":"ContainerDied","Data":"1d8bddd2e248539e1af43882fb53681d9dd509f9570ec2e3cef6b6b0f42586dd"}
Nov 02 21:10:26.279400 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.279285    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6" event={"ID":"f67432d1-22af-458b-bd57-002a02c62a5d","Type":"ContainerDied","Data":"78354821ec14decc0d7ff2ec9a9d627c4ed67a4e5ff7d20bd2a896e1f1f12f0b"}
Nov 02 21:10:26.279400 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.279309    5224 scope.go:122] "RemoveContainer" containerID="1d8bddd2e248539e1af43882fb53681d9dd509f9570ec2e3cef6b6b0f42586dd"
Nov 02 21:10:26.289802 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.289778    5224 scope.go:122] "RemoveContainer" containerID="1d8bddd2e248539e1af43882fb53681d9dd509f9570ec2e3cef6b6b0f42586dd"
Nov 02 21:10:26.290096 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:26.290071    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8bddd2e248539e1af43882fb53681d9dd509f9570ec2e3cef6b6b0f42586dd\": container with ID starting with 1d8bddd2e248539e1af43882fb53681d9dd509f9570ec2e3cef6b6b0f42586dd not found: ID does not exist" containerID="1d8bddd2e248539e1af43882fb53681d9dd509f9570ec2e3cef6b6b0f42586dd"
Nov 02 21:10:26.290168 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.290107    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8bddd2e248539e1af43882fb53681d9dd509f9570ec2e3cef6b6b0f42586dd"} err="failed to get container status \"1d8bddd2e248539e1af43882fb53681d9dd509f9570ec2e3cef6b6b0f42586dd\": rpc error: code = NotFound desc = could not find container \"1d8bddd2e248539e1af43882fb53681d9dd509f9570ec2e3cef6b6b0f42586dd\": container with ID starting with 1d8bddd2e248539e1af43882fb53681d9dd509f9570ec2e3cef6b6b0f42586dd not found: ID does not exist"
Nov 02 21:10:26.328197 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.328116    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67432d1-22af-458b-bd57-002a02c62a5d-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:26.328197 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.328157    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9995f\" (UniqueName: \"kubernetes.io/projected/f67432d1-22af-458b-bd57-002a02c62a5d-kube-api-access-9995f\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:26.328197 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.328175    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67432d1-22af-458b-bd57-002a02c62a5d-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:26.446163 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.446121    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"]
Nov 02 21:10:26.451295 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:26.451259    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-426941/simulator-w2-426941-kserve-7795cc9fdf-gvdg6"]
Nov 02 21:10:27.167960 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:27.167926    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67432d1-22af-458b-bd57-002a02c62a5d" path="/var/lib/kubelet/pods/f67432d1-22af-458b-bd57-002a02c62a5d/volumes"
Nov 02 21:10:27.359663 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:27.359628    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:10:29.859811 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:29.859749    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:30.095666 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:30.095628    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:33.003279 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:33.003215    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:33.242158 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:33.242116    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:34.334052 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:34.334013    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:34.596542 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:34.596449    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:35.042417 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:35.042380    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"]
Nov 02 21:10:35.042639 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:35.042613    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" podUID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" containerName="istio-proxy" containerID="cri-o://5ff6edc901f6778d89f1248ac472d5e86e58ae6ca1c1376427c33bbbf473b47c" gracePeriod=30
Nov 02 21:10:35.064351 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:35.064315    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:35.491344 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:35.491243    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:36.338046 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:36.337985    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:36.848527 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:36.848488    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-59gzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:10:36.848951 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:36.848569    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" podUID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:10:40.314519 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.314446    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 21:10:40.374434 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.374399    5224 generic.go:356] "Generic (PLEG): container finished" podID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" containerID="5ff6edc901f6778d89f1248ac472d5e86e58ae6ca1c1376427c33bbbf473b47c" exitCode=0
Nov 02 21:10:40.374615 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.374448    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"
Nov 02 21:10:40.374615 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.374479    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" event={"ID":"695b48a2-b9f5-4cb9-8e40-26835cdb5d71","Type":"ContainerDied","Data":"5ff6edc901f6778d89f1248ac472d5e86e58ae6ca1c1376427c33bbbf473b47c"}
Nov 02 21:10:40.374615 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.374535    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc" event={"ID":"695b48a2-b9f5-4cb9-8e40-26835cdb5d71","Type":"ContainerDied","Data":"c863d657db4335e27a5c2166a7d49359cd3145dccc5cd1a338a0b24d99761d9b"}
Nov 02 21:10:40.374615 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.374560    5224 scope.go:122] "RemoveContainer" containerID="5ff6edc901f6778d89f1248ac472d5e86e58ae6ca1c1376427c33bbbf473b47c"
Nov 02 21:10:40.384296 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.384238    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-certs\") pod \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") "
Nov 02 21:10:40.384390 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.384292    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istiod-ca-cert\") pod \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") "
Nov 02 21:10:40.384390 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.384320    5224 scope.go:122] "RemoveContainer" containerID="5ff6edc901f6778d89f1248ac472d5e86e58ae6ca1c1376427c33bbbf473b47c"
Nov 02 21:10:40.384390 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.384345    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-data\") pod \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") "
Nov 02 21:10:40.384390 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.384381    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-socket\") pod \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") "
Nov 02 21:10:40.384602 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.384426    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-kube-api-access-pmxld\" (UniqueName: \"kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-kube-api-access-pmxld\") pod \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") "
Nov 02 21:10:40.384602 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.384468    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-credential-socket\") pod \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") "
Nov 02 21:10:40.384602 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.384505    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-certs" pod "695b48a2-b9f5-4cb9-8e40-26835cdb5d71" (UID: "695b48a2-b9f5-4cb9-8e40-26835cdb5d71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:40.384602 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.384531    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-podinfo\") pod \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") "
Nov 02 21:10:40.384602 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.384573    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-token\" (UniqueName: \"kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-token\") pod \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") "
Nov 02 21:10:40.384917 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.384716    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-envoy\") pod \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\" (UID: \"695b48a2-b9f5-4cb9-8e40-26835cdb5d71\") "
Nov 02 21:10:40.384917 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:40.384805    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff6edc901f6778d89f1248ac472d5e86e58ae6ca1c1376427c33bbbf473b47c\": container with ID starting with 5ff6edc901f6778d89f1248ac472d5e86e58ae6ca1c1376427c33bbbf473b47c not found: ID does not exist" containerID="5ff6edc901f6778d89f1248ac472d5e86e58ae6ca1c1376427c33bbbf473b47c"
Nov 02 21:10:40.384917 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.384846    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff6edc901f6778d89f1248ac472d5e86e58ae6ca1c1376427c33bbbf473b47c"} err="failed to get container status \"5ff6edc901f6778d89f1248ac472d5e86e58ae6ca1c1376427c33bbbf473b47c\": rpc error: code = NotFound desc = could not find container \"5ff6edc901f6778d89f1248ac472d5e86e58ae6ca1c1376427c33bbbf473b47c\": container with ID starting with 5ff6edc901f6778d89f1248ac472d5e86e58ae6ca1c1376427c33bbbf473b47c not found: ID does not exist"
Nov 02 21:10:40.385042 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.384970    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istiod-ca-cert" pod "695b48a2-b9f5-4cb9-8e40-26835cdb5d71" (UID: "695b48a2-b9f5-4cb9-8e40-26835cdb5d71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:10:40.385276 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.385250    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-data" pod "695b48a2-b9f5-4cb9-8e40-26835cdb5d71" (UID: "695b48a2-b9f5-4cb9-8e40-26835cdb5d71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:40.385276 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.385267    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-credential-socket" pod "695b48a2-b9f5-4cb9-8e40-26835cdb5d71" (UID: "695b48a2-b9f5-4cb9-8e40-26835cdb5d71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:40.385432 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.385379    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:40.385432 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.385404    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:40.385512 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.385445    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-socket" pod "695b48a2-b9f5-4cb9-8e40-26835cdb5d71" (UID: "695b48a2-b9f5-4cb9-8e40-26835cdb5d71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:40.387321 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.387292    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-envoy" pod "695b48a2-b9f5-4cb9-8e40-26835cdb5d71" (UID: "695b48a2-b9f5-4cb9-8e40-26835cdb5d71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:40.387420 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.387385    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-podinfo" pod "695b48a2-b9f5-4cb9-8e40-26835cdb5d71" (UID: "695b48a2-b9f5-4cb9-8e40-26835cdb5d71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:10:40.387523 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.387505    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-kube-api-access-pmxld" pod "695b48a2-b9f5-4cb9-8e40-26835cdb5d71" (UID: "695b48a2-b9f5-4cb9-8e40-26835cdb5d71"). InnerVolumeSpecName "kube-api-access-pmxld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:10:40.387562 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.387543    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-token" pod "695b48a2-b9f5-4cb9-8e40-26835cdb5d71" (UID: "695b48a2-b9f5-4cb9-8e40-26835cdb5d71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:10:40.421671 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.421638    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"]
Nov 02 21:10:40.423931 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.423905    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"]
Nov 02 21:10:40.425841 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.425811    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"]
Nov 02 21:10:40.473070 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.473030    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"]
Nov 02 21:10:40.486257 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.486223    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:40.486257 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.486252    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:40.486257 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.486263    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:40.486506 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.486272    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmxld\" (UniqueName: \"kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-kube-api-access-pmxld\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:40.486506 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.486285    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:40.486506 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.486294    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:40.486506 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.486302    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695b48a2-b9f5-4cb9-8e40-26835cdb5d71-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:40.582529 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:40.582436    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:41.119953 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:41.119921    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"]
Nov 02 21:10:41.122811 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:41.122782    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-59gzc"]
Nov 02 21:10:41.167652 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:41.167620    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" path="/var/lib/kubelet/pods/695b48a2-b9f5-4cb9-8e40-26835cdb5d71/volumes"
Nov 02 21:10:41.520392 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:41.520356    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:42.577783 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:42.577725    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:44.285580 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:44.285544    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:44.472894 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:44.472860    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:46.431887 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:46.431848    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:48.736987 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:48.736699    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:48.921331 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:48.921289    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:49.169709 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:49.169664    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:49.354610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:49.354573    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:50.606126 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:50.606091    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:51.033753 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:51.033718    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"]
Nov 02 21:10:51.033987 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:51.033964    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" podUID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" containerName="istio-proxy" containerID="cri-o://3b5c5d1b38bfcf72be4b7868a46250b5a1f37ec91a1578c4904326ddcc29a4f1" gracePeriod=30
Nov 02 21:10:51.074586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:51.074537    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:51.405792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:51.405685    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:52.259750 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:52.259716    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:52.475242 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:52.475206    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:54.422428 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:54.422384    5224 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:10:54.422859 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:54.422455    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" podUID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:10:54.668090 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:54.668056    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:54.854207 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:54.854165    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:55.051020 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.050976    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:55.442387 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.442348    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd" podUID="bc2af629-50fa-4966-87d5-d4d399d844b1" containerName="main" containerID="cri-o://f6a8b16bcaad2ab0937bd1855b1ce273df18b2a82debacdd4c92a58ddca52e66" gracePeriod=45
Nov 02 21:10:55.444337 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.444303    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg" podUID="c2b4bca4-928d-439e-b582-979ff1768fc6" containerName="main" containerID="cri-o://2235f79493f9f3c820b51149a6688b716eae5dd107c19c9ffa85404c9b0482cf" gracePeriod=45
Nov 02 21:10:55.445659 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.445629    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" podUID="87bb590b-e94c-4051-9df7-09be7673c1e8" containerName="main" containerID="cri-o://8502226f1e020a1490dd1690cc7cf28e553c91912279059b608110e2e278d167" gracePeriod=45
Nov 02 21:10:55.494708 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.494670    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerName="main" containerID="cri-o://3a34d3737cd7f163cfd09a3f2f0adacecbded81af8fd2a27232e97bb134c887e" gracePeriod=45
Nov 02 21:10:55.677725 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.677024    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" podUID="87bb590b-e94c-4051-9df7-09be7673c1e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": dial tcp 10.134.0.51:8000: connect: connection refused"
Nov 02 21:10:55.798697 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.798661    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 21:10:55.800747 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.800720    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 21:10:55.821259 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.821229    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 21:10:55.824261 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.824237    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 21:10:55.878306 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878269    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-tmp-dir\") pod \"c2b4bca4-928d-439e-b582-979ff1768fc6\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") "
Nov 02 21:10:55.878499 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878316    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87bb590b-e94c-4051-9df7-09be7673c1e8-kube-api-access-qp8dc\" (UniqueName: \"kubernetes.io/projected/87bb590b-e94c-4051-9df7-09be7673c1e8-kube-api-access-qp8dc\") pod \"87bb590b-e94c-4051-9df7-09be7673c1e8\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") "
Nov 02 21:10:55.878499 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878339    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-model-cache\") pod \"c2b4bca4-928d-439e-b582-979ff1768fc6\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") "
Nov 02 21:10:55.878499 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878380    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b4bca4-928d-439e-b582-979ff1768fc6-kube-api-access-sftm5\" (UniqueName: \"kubernetes.io/projected/c2b4bca4-928d-439e-b582-979ff1768fc6-kube-api-access-sftm5\") pod \"c2b4bca4-928d-439e-b582-979ff1768fc6\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") "
Nov 02 21:10:55.878499 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878398    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-dshm\") pod \"87bb590b-e94c-4051-9df7-09be7673c1e8\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") "
Nov 02 21:10:55.878499 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878417    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-home\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-home\") pod \"87bb590b-e94c-4051-9df7-09be7673c1e8\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") "
Nov 02 21:10:55.878499 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878445    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87bb590b-e94c-4051-9df7-09be7673c1e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/87bb590b-e94c-4051-9df7-09be7673c1e8-tls-certs\") pod \"87bb590b-e94c-4051-9df7-09be7673c1e8\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") "
Nov 02 21:10:55.878499 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878486    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-dshm\") pod \"c2b4bca4-928d-439e-b582-979ff1768fc6\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") "
Nov 02 21:10:55.878952 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878571    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-home\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-home\") pod \"c2b4bca4-928d-439e-b582-979ff1768fc6\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") "
Nov 02 21:10:55.878952 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878626    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-model-cache\") pod \"87bb590b-e94c-4051-9df7-09be7673c1e8\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") "
Nov 02 21:10:55.878952 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878615    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-tmp-dir" pod "c2b4bca4-928d-439e-b582-979ff1768fc6" (UID: "c2b4bca4-928d-439e-b582-979ff1768fc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.878952 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878672    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-tmp-dir\") pod \"87bb590b-e94c-4051-9df7-09be7673c1e8\" (UID: \"87bb590b-e94c-4051-9df7-09be7673c1e8\") "
Nov 02 21:10:55.878952 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878700    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-model-cache" pod "c2b4bca4-928d-439e-b582-979ff1768fc6" (UID: "c2b4bca4-928d-439e-b582-979ff1768fc6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.878952 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878807    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2b4bca4-928d-439e-b582-979ff1768fc6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b4bca4-928d-439e-b582-979ff1768fc6-tls-certs\") pod \"c2b4bca4-928d-439e-b582-979ff1768fc6\" (UID: \"c2b4bca4-928d-439e-b582-979ff1768fc6\") "
Nov 02 21:10:55.878952 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.878902    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-home" pod "c2b4bca4-928d-439e-b582-979ff1768fc6" (UID: "c2b4bca4-928d-439e-b582-979ff1768fc6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.879334 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.879132    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-home" pod "87bb590b-e94c-4051-9df7-09be7673c1e8" (UID: "87bb590b-e94c-4051-9df7-09be7673c1e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.879395 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.879360    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-tmp-dir" pod "87bb590b-e94c-4051-9df7-09be7673c1e8" (UID: "87bb590b-e94c-4051-9df7-09be7673c1e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.879453 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.879393    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-model-cache" pod "87bb590b-e94c-4051-9df7-09be7673c1e8" (UID: "87bb590b-e94c-4051-9df7-09be7673c1e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.879515 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.879495    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.879573 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.879515    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.879573 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.879530    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.879573 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.879546    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.881446 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.881402    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-dshm" pod "c2b4bca4-928d-439e-b582-979ff1768fc6" (UID: "c2b4bca4-928d-439e-b582-979ff1768fc6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.881446 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.881425    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-dshm" pod "87bb590b-e94c-4051-9df7-09be7673c1e8" (UID: "87bb590b-e94c-4051-9df7-09be7673c1e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.881598 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.881463    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87bb590b-e94c-4051-9df7-09be7673c1e8-kube-api-access-qp8dc" pod "87bb590b-e94c-4051-9df7-09be7673c1e8" (UID: "87bb590b-e94c-4051-9df7-09be7673c1e8"). InnerVolumeSpecName "kube-api-access-qp8dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:10:55.881598 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.881470    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b4bca4-928d-439e-b582-979ff1768fc6-tls-certs" pod "c2b4bca4-928d-439e-b582-979ff1768fc6" (UID: "c2b4bca4-928d-439e-b582-979ff1768fc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:10:55.881598 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.881487    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b4bca4-928d-439e-b582-979ff1768fc6-kube-api-access-sftm5" pod "c2b4bca4-928d-439e-b582-979ff1768fc6" (UID: "c2b4bca4-928d-439e-b582-979ff1768fc6"). InnerVolumeSpecName "kube-api-access-sftm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:10:55.881598 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.881495    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87bb590b-e94c-4051-9df7-09be7673c1e8-tls-certs" pod "87bb590b-e94c-4051-9df7-09be7673c1e8" (UID: "87bb590b-e94c-4051-9df7-09be7673c1e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:10:55.980635 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.980554    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc2af629-50fa-4966-87d5-d4d399d844b1-kube-api-access-6q55m\" (UniqueName: \"kubernetes.io/projected/bc2af629-50fa-4966-87d5-d4d399d844b1-kube-api-access-6q55m\") pod \"bc2af629-50fa-4966-87d5-d4d399d844b1\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") "
Nov 02 21:10:55.980635 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.980590    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa503409-81d4-4532-9dc8-64a7aa8d0443-kube-api-access-pcc6s\" (UniqueName: \"kubernetes.io/projected/aa503409-81d4-4532-9dc8-64a7aa8d0443-kube-api-access-pcc6s\") pod \"aa503409-81d4-4532-9dc8-64a7aa8d0443\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") "
Nov 02 21:10:55.980635 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.980627    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-dshm\") pod \"aa503409-81d4-4532-9dc8-64a7aa8d0443\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") "
Nov 02 21:10:55.980934 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.980697    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-tmp-dir\") pod \"bc2af629-50fa-4966-87d5-d4d399d844b1\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") "
Nov 02 21:10:55.980934 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.980743    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-dshm\") pod \"bc2af629-50fa-4966-87d5-d4d399d844b1\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") "
Nov 02 21:10:55.980934 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.980787    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-model-cache\") pod \"aa503409-81d4-4532-9dc8-64a7aa8d0443\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") "
Nov 02 21:10:55.980934 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.980820    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-home\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-home\") pod \"aa503409-81d4-4532-9dc8-64a7aa8d0443\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") "
Nov 02 21:10:55.980934 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.980869    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-home\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-home\") pod \"bc2af629-50fa-4966-87d5-d4d399d844b1\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") "
Nov 02 21:10:55.980934 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.980928    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa503409-81d4-4532-9dc8-64a7aa8d0443-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa503409-81d4-4532-9dc8-64a7aa8d0443-tls-certs\") pod \"aa503409-81d4-4532-9dc8-64a7aa8d0443\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") "
Nov 02 21:10:55.981288 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.980978    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-model-cache\") pod \"bc2af629-50fa-4966-87d5-d4d399d844b1\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") "
Nov 02 21:10:55.981288 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981008    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc2af629-50fa-4966-87d5-d4d399d844b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2af629-50fa-4966-87d5-d4d399d844b1-tls-certs\") pod \"bc2af629-50fa-4966-87d5-d4d399d844b1\" (UID: \"bc2af629-50fa-4966-87d5-d4d399d844b1\") "
Nov 02 21:10:55.981288 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981017    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-tmp-dir" pod "bc2af629-50fa-4966-87d5-d4d399d844b1" (UID: "bc2af629-50fa-4966-87d5-d4d399d844b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.981288 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981032    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-model-cache" pod "aa503409-81d4-4532-9dc8-64a7aa8d0443" (UID: "aa503409-81d4-4532-9dc8-64a7aa8d0443"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.981288 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981069    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-tmp-dir\") pod \"aa503409-81d4-4532-9dc8-64a7aa8d0443\" (UID: \"aa503409-81d4-4532-9dc8-64a7aa8d0443\") "
Nov 02 21:10:55.981288 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981234    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-home" pod "bc2af629-50fa-4966-87d5-d4d399d844b1" (UID: "bc2af629-50fa-4966-87d5-d4d399d844b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.981288 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981238    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-home" pod "aa503409-81d4-4532-9dc8-64a7aa8d0443" (UID: "aa503409-81d4-4532-9dc8-64a7aa8d0443"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.981686 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981447    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-model-cache" pod "bc2af629-50fa-4966-87d5-d4d399d844b1" (UID: "bc2af629-50fa-4966-87d5-d4d399d844b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.981686 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981561    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-tmp-dir" pod "aa503409-81d4-4532-9dc8-64a7aa8d0443" (UID: "aa503409-81d4-4532-9dc8-64a7aa8d0443"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.981686 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981635    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.981686 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981655    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.981686 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981673    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sftm5\" (UniqueName: \"kubernetes.io/projected/c2b4bca4-928d-439e-b582-979ff1768fc6-kube-api-access-sftm5\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.981990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981690    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.981990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981704    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.981990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981719    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87bb590b-e94c-4051-9df7-09be7673c1e8-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.981990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981732    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b4bca4-928d-439e-b582-979ff1768fc6-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.981990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981746    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.981990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981759    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.981990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981802    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87bb590b-e94c-4051-9df7-09be7673c1e8-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.981990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981815    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b4bca4-928d-439e-b582-979ff1768fc6-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.981990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981828    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.981990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.981843    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp8dc\" (UniqueName: \"kubernetes.io/projected/87bb590b-e94c-4051-9df7-09be7673c1e8-kube-api-access-qp8dc\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:55.983068 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.983044    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2af629-50fa-4966-87d5-d4d399d844b1-kube-api-access-6q55m" pod "bc2af629-50fa-4966-87d5-d4d399d844b1" (UID: "bc2af629-50fa-4966-87d5-d4d399d844b1"). InnerVolumeSpecName "kube-api-access-6q55m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:10:55.983309 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.983278    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa503409-81d4-4532-9dc8-64a7aa8d0443-kube-api-access-pcc6s" pod "aa503409-81d4-4532-9dc8-64a7aa8d0443" (UID: "aa503409-81d4-4532-9dc8-64a7aa8d0443"). InnerVolumeSpecName "kube-api-access-pcc6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:10:55.983309 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.983284    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-dshm" pod "aa503409-81d4-4532-9dc8-64a7aa8d0443" (UID: "aa503409-81d4-4532-9dc8-64a7aa8d0443"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.983499 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.983432    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-dshm" pod "bc2af629-50fa-4966-87d5-d4d399d844b1" (UID: "bc2af629-50fa-4966-87d5-d4d399d844b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:55.983576 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.983549    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa503409-81d4-4532-9dc8-64a7aa8d0443-tls-certs" pod "aa503409-81d4-4532-9dc8-64a7aa8d0443" (UID: "aa503409-81d4-4532-9dc8-64a7aa8d0443"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:10:55.983735 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:55.983717    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc2af629-50fa-4966-87d5-d4d399d844b1-tls-certs" pod "bc2af629-50fa-4966-87d5-d4d399d844b1" (UID: "bc2af629-50fa-4966-87d5-d4d399d844b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:10:56.082957 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.082922    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.082957 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.082952    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q55m\" (UniqueName: \"kubernetes.io/projected/bc2af629-50fa-4966-87d5-d4d399d844b1-kube-api-access-6q55m\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.082957 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.082963    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcc6s\" (UniqueName: \"kubernetes.io/projected/aa503409-81d4-4532-9dc8-64a7aa8d0443-kube-api-access-pcc6s\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.083195 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.082971    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa503409-81d4-4532-9dc8-64a7aa8d0443-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.083195 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.082979    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc2af629-50fa-4966-87d5-d4d399d844b1-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.083195 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.082987    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa503409-81d4-4532-9dc8-64a7aa8d0443-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.083195 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.082996    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2af629-50fa-4966-87d5-d4d399d844b1-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.280830 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.280738    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 21:10:56.387358 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.387323    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-envoy\") pod \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") "
Nov 02 21:10:56.387551 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.387387    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-token\" (UniqueName: \"kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-token\") pod \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") "
Nov 02 21:10:56.387551 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.387442    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-credential-socket\") pod \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") "
Nov 02 21:10:56.387551 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.387506    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-kube-api-access-bd8n6\" (UniqueName: \"kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-kube-api-access-bd8n6\") pod \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") "
Nov 02 21:10:56.387676 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.387578    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-podinfo\") pod \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") "
Nov 02 21:10:56.387676 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.387605    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-socket\") pod \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") "
Nov 02 21:10:56.387813 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.387697    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-certs\") pod \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") "
Nov 02 21:10:56.387813 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.387733    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-data\") pod \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") "
Nov 02 21:10:56.387813 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.387746    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-credential-socket" pod "d3f254db-a4e4-4fed-ac7d-d8b61600091e" (UID: "d3f254db-a4e4-4fed-ac7d-d8b61600091e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:56.387987 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.387816    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istiod-ca-cert\") pod \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\" (UID: \"d3f254db-a4e4-4fed-ac7d-d8b61600091e\") "
Nov 02 21:10:56.387987 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.387923    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-socket" pod "d3f254db-a4e4-4fed-ac7d-d8b61600091e" (UID: "d3f254db-a4e4-4fed-ac7d-d8b61600091e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:56.388109 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.388001    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-certs" pod "d3f254db-a4e4-4fed-ac7d-d8b61600091e" (UID: "d3f254db-a4e4-4fed-ac7d-d8b61600091e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:56.388175 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.388142    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-data" pod "d3f254db-a4e4-4fed-ac7d-d8b61600091e" (UID: "d3f254db-a4e4-4fed-ac7d-d8b61600091e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:56.388341 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.388316    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istiod-ca-cert" pod "d3f254db-a4e4-4fed-ac7d-d8b61600091e" (UID: "d3f254db-a4e4-4fed-ac7d-d8b61600091e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:10:56.388430 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.388328    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.388430 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.388375    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.388430 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.388391    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.388430 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.388406    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.389643 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.389618    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-envoy" pod "d3f254db-a4e4-4fed-ac7d-d8b61600091e" (UID: "d3f254db-a4e4-4fed-ac7d-d8b61600091e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:56.389736 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.389702    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-podinfo" pod "d3f254db-a4e4-4fed-ac7d-d8b61600091e" (UID: "d3f254db-a4e4-4fed-ac7d-d8b61600091e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:10:56.390018 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.389999    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-kube-api-access-bd8n6" pod "d3f254db-a4e4-4fed-ac7d-d8b61600091e" (UID: "d3f254db-a4e4-4fed-ac7d-d8b61600091e"). InnerVolumeSpecName "kube-api-access-bd8n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:10:56.390085 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.390011    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-token" pod "d3f254db-a4e4-4fed-ac7d-d8b61600091e" (UID: "d3f254db-a4e4-4fed-ac7d-d8b61600091e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:10:56.477378 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.477343    5224 generic.go:356] "Generic (PLEG): container finished" podID="c2b4bca4-928d-439e-b582-979ff1768fc6" containerID="2235f79493f9f3c820b51149a6688b716eae5dd107c19c9ffa85404c9b0482cf" exitCode=0
Nov 02 21:10:56.477820 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.477388    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"
Nov 02 21:10:56.477820 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.477431    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg" event={"ID":"c2b4bca4-928d-439e-b582-979ff1768fc6","Type":"ContainerDied","Data":"2235f79493f9f3c820b51149a6688b716eae5dd107c19c9ffa85404c9b0482cf"}
Nov 02 21:10:56.477820 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.477483    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg" event={"ID":"c2b4bca4-928d-439e-b582-979ff1768fc6","Type":"ContainerDied","Data":"99ca4c2b73e8a032e35ea680d56cf1943ce869774c6c73c4f3ef8c49b64f7f9e"}
Nov 02 21:10:56.477820 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.477507    5224 scope.go:122] "RemoveContainer" containerID="2235f79493f9f3c820b51149a6688b716eae5dd107c19c9ffa85404c9b0482cf"
Nov 02 21:10:56.478926 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.478905    5224 generic.go:356] "Generic (PLEG): container finished" podID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" containerID="3b5c5d1b38bfcf72be4b7868a46250b5a1f37ec91a1578c4904326ddcc29a4f1" exitCode=0
Nov 02 21:10:56.479012 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.478980    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"
Nov 02 21:10:56.479069 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.479048    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" event={"ID":"d3f254db-a4e4-4fed-ac7d-d8b61600091e","Type":"ContainerDied","Data":"3b5c5d1b38bfcf72be4b7868a46250b5a1f37ec91a1578c4904326ddcc29a4f1"}
Nov 02 21:10:56.479113 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.479089    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt" event={"ID":"d3f254db-a4e4-4fed-ac7d-d8b61600091e","Type":"ContainerDied","Data":"47d62c5ecf9cca11216f8ac234bf93bac6304fc055a1566f1dc25a21d5c6bf66"}
Nov 02 21:10:56.480601 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.480579    5224 generic.go:356] "Generic (PLEG): container finished" podID="bc2af629-50fa-4966-87d5-d4d399d844b1" containerID="f6a8b16bcaad2ab0937bd1855b1ce273df18b2a82debacdd4c92a58ddca52e66" exitCode=0
Nov 02 21:10:56.480722 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.480614    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"
Nov 02 21:10:56.480722 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.480690    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd" event={"ID":"bc2af629-50fa-4966-87d5-d4d399d844b1","Type":"ContainerDied","Data":"f6a8b16bcaad2ab0937bd1855b1ce273df18b2a82debacdd4c92a58ddca52e66"}
Nov 02 21:10:56.480854 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.480730    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd" event={"ID":"bc2af629-50fa-4966-87d5-d4d399d844b1","Type":"ContainerDied","Data":"885b114f1c7194993fd887ac81d6f57331dad5df5b15759a47543350501a41f4"}
Nov 02 21:10:56.484273 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.484251    5224 generic.go:356] "Generic (PLEG): container finished" podID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerID="3a34d3737cd7f163cfd09a3f2f0adacecbded81af8fd2a27232e97bb134c887e" exitCode=0
Nov 02 21:10:56.484390 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.484299    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"
Nov 02 21:10:56.484390 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.484338    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" event={"ID":"aa503409-81d4-4532-9dc8-64a7aa8d0443","Type":"ContainerDied","Data":"3a34d3737cd7f163cfd09a3f2f0adacecbded81af8fd2a27232e97bb134c887e"}
Nov 02 21:10:56.484390 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.484373    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng" event={"ID":"aa503409-81d4-4532-9dc8-64a7aa8d0443","Type":"ContainerDied","Data":"50e7d4a17d0d0801e4ce605fd1fe88c280f2114f34481d9af455368ddb0a2d45"}
Nov 02 21:10:56.485943 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.485825    5224 generic.go:356] "Generic (PLEG): container finished" podID="87bb590b-e94c-4051-9df7-09be7673c1e8" containerID="8502226f1e020a1490dd1690cc7cf28e553c91912279059b608110e2e278d167" exitCode=0
Nov 02 21:10:56.485943 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.485874    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"
Nov 02 21:10:56.485943 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.485893    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" event={"ID":"87bb590b-e94c-4051-9df7-09be7673c1e8","Type":"ContainerDied","Data":"8502226f1e020a1490dd1690cc7cf28e553c91912279059b608110e2e278d167"}
Nov 02 21:10:56.485943 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.485918    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw" event={"ID":"87bb590b-e94c-4051-9df7-09be7673c1e8","Type":"ContainerDied","Data":"d2c0c8335f1164328d1c1112ca442c69a75fc7012d09ad1505bd2c43edee2565"}
Nov 02 21:10:56.488838 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.488820    5224 scope.go:122] "RemoveContainer" containerID="2235f79493f9f3c820b51149a6688b716eae5dd107c19c9ffa85404c9b0482cf"
Nov 02 21:10:56.489239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.489088    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.489239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.489135    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd8n6\" (UniqueName: \"kubernetes.io/projected/d3f254db-a4e4-4fed-ac7d-d8b61600091e-kube-api-access-bd8n6\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.489239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.489153    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.489239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.489183    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.489239 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:56.489130    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2235f79493f9f3c820b51149a6688b716eae5dd107c19c9ffa85404c9b0482cf\": container with ID starting with 2235f79493f9f3c820b51149a6688b716eae5dd107c19c9ffa85404c9b0482cf not found: ID does not exist" containerID="2235f79493f9f3c820b51149a6688b716eae5dd107c19c9ffa85404c9b0482cf"
Nov 02 21:10:56.489687 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.489253    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2235f79493f9f3c820b51149a6688b716eae5dd107c19c9ffa85404c9b0482cf"} err="failed to get container status \"2235f79493f9f3c820b51149a6688b716eae5dd107c19c9ffa85404c9b0482cf\": rpc error: code = NotFound desc = could not find container \"2235f79493f9f3c820b51149a6688b716eae5dd107c19c9ffa85404c9b0482cf\": container with ID starting with 2235f79493f9f3c820b51149a6688b716eae5dd107c19c9ffa85404c9b0482cf not found: ID does not exist"
Nov 02 21:10:56.489687 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.489279    5224 scope.go:122] "RemoveContainer" containerID="3b5c5d1b38bfcf72be4b7868a46250b5a1f37ec91a1578c4904326ddcc29a4f1"
Nov 02 21:10:56.489687 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.489228    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f254db-a4e4-4fed-ac7d-d8b61600091e-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:56.500335 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.500308    5224 scope.go:122] "RemoveContainer" containerID="3b5c5d1b38bfcf72be4b7868a46250b5a1f37ec91a1578c4904326ddcc29a4f1"
Nov 02 21:10:56.500643 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:56.500617    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5c5d1b38bfcf72be4b7868a46250b5a1f37ec91a1578c4904326ddcc29a4f1\": container with ID starting with 3b5c5d1b38bfcf72be4b7868a46250b5a1f37ec91a1578c4904326ddcc29a4f1 not found: ID does not exist" containerID="3b5c5d1b38bfcf72be4b7868a46250b5a1f37ec91a1578c4904326ddcc29a4f1"
Nov 02 21:10:56.500716 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.500656    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5c5d1b38bfcf72be4b7868a46250b5a1f37ec91a1578c4904326ddcc29a4f1"} err="failed to get container status \"3b5c5d1b38bfcf72be4b7868a46250b5a1f37ec91a1578c4904326ddcc29a4f1\": rpc error: code = NotFound desc = could not find container \"3b5c5d1b38bfcf72be4b7868a46250b5a1f37ec91a1578c4904326ddcc29a4f1\": container with ID starting with 3b5c5d1b38bfcf72be4b7868a46250b5a1f37ec91a1578c4904326ddcc29a4f1 not found: ID does not exist"
Nov 02 21:10:56.500716 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.500680    5224 scope.go:122] "RemoveContainer" containerID="f6a8b16bcaad2ab0937bd1855b1ce273df18b2a82debacdd4c92a58ddca52e66"
Nov 02 21:10:56.509456 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.509435    5224 scope.go:122] "RemoveContainer" containerID="f6a8b16bcaad2ab0937bd1855b1ce273df18b2a82debacdd4c92a58ddca52e66"
Nov 02 21:10:56.509704 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:56.509681    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a8b16bcaad2ab0937bd1855b1ce273df18b2a82debacdd4c92a58ddca52e66\": container with ID starting with f6a8b16bcaad2ab0937bd1855b1ce273df18b2a82debacdd4c92a58ddca52e66 not found: ID does not exist" containerID="f6a8b16bcaad2ab0937bd1855b1ce273df18b2a82debacdd4c92a58ddca52e66"
Nov 02 21:10:56.509862 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.509712    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a8b16bcaad2ab0937bd1855b1ce273df18b2a82debacdd4c92a58ddca52e66"} err="failed to get container status \"f6a8b16bcaad2ab0937bd1855b1ce273df18b2a82debacdd4c92a58ddca52e66\": rpc error: code = NotFound desc = could not find container \"f6a8b16bcaad2ab0937bd1855b1ce273df18b2a82debacdd4c92a58ddca52e66\": container with ID starting with f6a8b16bcaad2ab0937bd1855b1ce273df18b2a82debacdd4c92a58ddca52e66 not found: ID does not exist"
Nov 02 21:10:56.509862 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.509731    5224 scope.go:122] "RemoveContainer" containerID="3a34d3737cd7f163cfd09a3f2f0adacecbded81af8fd2a27232e97bb134c887e"
Nov 02 21:10:56.517688 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.517672    5224 scope.go:122] "RemoveContainer" containerID="3a34d3737cd7f163cfd09a3f2f0adacecbded81af8fd2a27232e97bb134c887e"
Nov 02 21:10:56.517983 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:56.517966    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a34d3737cd7f163cfd09a3f2f0adacecbded81af8fd2a27232e97bb134c887e\": container with ID starting with 3a34d3737cd7f163cfd09a3f2f0adacecbded81af8fd2a27232e97bb134c887e not found: ID does not exist" containerID="3a34d3737cd7f163cfd09a3f2f0adacecbded81af8fd2a27232e97bb134c887e"
Nov 02 21:10:56.518065 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.517994    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a34d3737cd7f163cfd09a3f2f0adacecbded81af8fd2a27232e97bb134c887e"} err="failed to get container status \"3a34d3737cd7f163cfd09a3f2f0adacecbded81af8fd2a27232e97bb134c887e\": rpc error: code = NotFound desc = could not find container \"3a34d3737cd7f163cfd09a3f2f0adacecbded81af8fd2a27232e97bb134c887e\": container with ID starting with 3a34d3737cd7f163cfd09a3f2f0adacecbded81af8fd2a27232e97bb134c887e not found: ID does not exist"
Nov 02 21:10:56.518065 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.518019    5224 scope.go:122] "RemoveContainer" containerID="8502226f1e020a1490dd1690cc7cf28e553c91912279059b608110e2e278d167"
Nov 02 21:10:56.525917 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.525900    5224 scope.go:122] "RemoveContainer" containerID="8502226f1e020a1490dd1690cc7cf28e553c91912279059b608110e2e278d167"
Nov 02 21:10:56.526199 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:56.526181    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8502226f1e020a1490dd1690cc7cf28e553c91912279059b608110e2e278d167\": container with ID starting with 8502226f1e020a1490dd1690cc7cf28e553c91912279059b608110e2e278d167 not found: ID does not exist" containerID="8502226f1e020a1490dd1690cc7cf28e553c91912279059b608110e2e278d167"
Nov 02 21:10:56.526262 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.526204    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8502226f1e020a1490dd1690cc7cf28e553c91912279059b608110e2e278d167"} err="failed to get container status \"8502226f1e020a1490dd1690cc7cf28e553c91912279059b608110e2e278d167\": rpc error: code = NotFound desc = could not find container \"8502226f1e020a1490dd1690cc7cf28e553c91912279059b608110e2e278d167\": container with ID starting with 8502226f1e020a1490dd1690cc7cf28e553c91912279059b608110e2e278d167 not found: ID does not exist"
Nov 02 21:10:56.573274 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.573240    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:10:56.643518 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.643479    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"]
Nov 02 21:10:56.646970 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.646939    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/distinct-w3-6a4ab0-kserve-57bff95d6b-k68pg"]
Nov 02 21:10:56.757962 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.757931    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"]
Nov 02 21:10:56.761137 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.761115    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/distinct-2-w3-6a4ab0-kserve-75dfbcd8f-cwwvd"]
Nov 02 21:10:56.872675 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.872590    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"]
Nov 02 21:10:56.879025 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.878998    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-8n6nt"]
Nov 02 21:10:56.988854 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.988817    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"]
Nov 02 21:10:56.992246 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:56.992221    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/simulator-w3-6a4ab0-kserve-756bd89d98-brcng"]
Nov 02 21:10:57.105857 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:57.105827    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"]
Nov 02 21:10:57.108969 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:57.108946    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6a4ab0/premium-w3-6a4ab0-kserve-6955c4bd79-jnvnw"]
Nov 02 21:10:57.167207 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:57.167122    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87bb590b-e94c-4051-9df7-09be7673c1e8" path="/var/lib/kubelet/pods/87bb590b-e94c-4051-9df7-09be7673c1e8/volumes"
Nov 02 21:10:57.167571 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:57.167557    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" path="/var/lib/kubelet/pods/aa503409-81d4-4532-9dc8-64a7aa8d0443/volumes"
Nov 02 21:10:57.167941 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:57.167928    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc2af629-50fa-4966-87d5-d4d399d844b1" path="/var/lib/kubelet/pods/bc2af629-50fa-4966-87d5-d4d399d844b1/volumes"
Nov 02 21:10:57.168588 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:57.168573    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b4bca4-928d-439e-b582-979ff1768fc6" path="/var/lib/kubelet/pods/c2b4bca4-928d-439e-b582-979ff1768fc6/volumes"
Nov 02 21:10:57.169022 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:57.169009    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" path="/var/lib/kubelet/pods/d3f254db-a4e4-4fed-ac7d-d8b61600091e/volumes"
Nov 02 21:10:57.358562 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:10:57.358522    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:10:58.938187 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:10:58.938148    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:01.078551 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:01.078514    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:01.551527 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:01.551489    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:06.445304 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:06.445258    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:10.768498 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:10.768454    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:12.510576 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:12.510536    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:12.735107 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:12.735073    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:13.429942 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:13.429901    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:15.056344 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:15.056304    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:15.287246 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:15.287213    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:16.948314 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:16.948274    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:17.468626 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:17.468577    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:18.017941 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:18.017898    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:18.217923 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:18.217827    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:18.742579 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:18.742545    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:19.976197 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:19.976162    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:20.204693 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:20.204659    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:20.370503 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:20.370472    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"]
Nov 02 21:11:20.370693 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:20.370668    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy" containerID="cri-o://78712418b6bf4ffd1d160982b72a164881906739011c2622413a536f41cace63" gracePeriod=30
Nov 02 21:11:20.983044 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:20.983009    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:21.459726 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:21.459682    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:22.311200 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:22.311155    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:11:22.311582 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:22.311230    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:11:22.567297 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:22.567205    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:24.442021 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:24.441991    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:25.627518 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.627491    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 21:11:25.656560 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.656489    5224 generic.go:356] "Generic (PLEG): container finished" podID="3542c449-0d85-4534-9284-67c7250ccd64" containerID="78712418b6bf4ffd1d160982b72a164881906739011c2622413a536f41cace63" exitCode=0
Nov 02 21:11:25.656560 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.656536    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"
Nov 02 21:11:25.656818 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.656573    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" event={"ID":"3542c449-0d85-4534-9284-67c7250ccd64","Type":"ContainerDied","Data":"78712418b6bf4ffd1d160982b72a164881906739011c2622413a536f41cace63"}
Nov 02 21:11:25.656818 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.656641    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5" event={"ID":"3542c449-0d85-4534-9284-67c7250ccd64","Type":"ContainerDied","Data":"562ea3a7dc37023cd28efa3316f24a26413cbedf4908e018b3576bd04e9905d3"}
Nov 02 21:11:25.656818 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.656673    5224 scope.go:122] "RemoveContainer" containerID="78712418b6bf4ffd1d160982b72a164881906739011c2622413a536f41cace63"
Nov 02 21:11:25.666171 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.666154    5224 scope.go:122] "RemoveContainer" containerID="78712418b6bf4ffd1d160982b72a164881906739011c2622413a536f41cace63"
Nov 02 21:11:25.666438 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:11:25.666418    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78712418b6bf4ffd1d160982b72a164881906739011c2622413a536f41cace63\": container with ID starting with 78712418b6bf4ffd1d160982b72a164881906739011c2622413a536f41cace63 not found: ID does not exist" containerID="78712418b6bf4ffd1d160982b72a164881906739011c2622413a536f41cace63"
Nov 02 21:11:25.666523 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.666452    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78712418b6bf4ffd1d160982b72a164881906739011c2622413a536f41cace63"} err="failed to get container status \"78712418b6bf4ffd1d160982b72a164881906739011c2622413a536f41cace63\": rpc error: code = NotFound desc = could not find container \"78712418b6bf4ffd1d160982b72a164881906739011c2622413a536f41cace63\": container with ID starting with 78712418b6bf4ffd1d160982b72a164881906739011c2622413a536f41cace63 not found: ID does not exist"
Nov 02 21:11:25.684878 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.684849    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-istio-token\" (UniqueName: \"kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-istio-token\") pod \"3542c449-0d85-4534-9284-67c7250ccd64\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") "
Nov 02 21:11:25.684987 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.684898    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3542c449-0d85-4534-9284-67c7250ccd64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3542c449-0d85-4534-9284-67c7250ccd64-istio-podinfo\") pod \"3542c449-0d85-4534-9284-67c7250ccd64\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") "
Nov 02 21:11:25.684987 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.684958    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-data\") pod \"3542c449-0d85-4534-9284-67c7250ccd64\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") "
Nov 02 21:11:25.684987 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.684975    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3542c449-0d85-4534-9284-67c7250ccd64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3542c449-0d85-4534-9284-67c7250ccd64-istiod-ca-cert\") pod \"3542c449-0d85-4534-9284-67c7250ccd64\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") "
Nov 02 21:11:25.685144 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685035    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-credential-socket\") pod \"3542c449-0d85-4534-9284-67c7250ccd64\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") "
Nov 02 21:11:25.685144 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685053    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-certs\") pod \"3542c449-0d85-4534-9284-67c7250ccd64\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") "
Nov 02 21:11:25.685144 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685086    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-socket\") pod \"3542c449-0d85-4534-9284-67c7250ccd64\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") "
Nov 02 21:11:25.685144 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685112    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-envoy\") pod \"3542c449-0d85-4534-9284-67c7250ccd64\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") "
Nov 02 21:11:25.685365 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685159    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-kube-api-access-9n4th\" (UniqueName: \"kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-kube-api-access-9n4th\") pod \"3542c449-0d85-4534-9284-67c7250ccd64\" (UID: \"3542c449-0d85-4534-9284-67c7250ccd64\") "
Nov 02 21:11:25.685365 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685206    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-data" pod "3542c449-0d85-4534-9284-67c7250ccd64" (UID: "3542c449-0d85-4534-9284-67c7250ccd64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:11:25.685365 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685304    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-credential-socket" pod "3542c449-0d85-4534-9284-67c7250ccd64" (UID: "3542c449-0d85-4534-9284-67c7250ccd64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:11:25.685517 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685392    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-socket" pod "3542c449-0d85-4534-9284-67c7250ccd64" (UID: "3542c449-0d85-4534-9284-67c7250ccd64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:11:25.685517 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685436    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3542c449-0d85-4534-9284-67c7250ccd64-istiod-ca-cert" pod "3542c449-0d85-4534-9284-67c7250ccd64" (UID: "3542c449-0d85-4534-9284-67c7250ccd64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:11:25.685671 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685652    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-certs" pod "3542c449-0d85-4534-9284-67c7250ccd64" (UID: "3542c449-0d85-4534-9284-67c7250ccd64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:11:25.685732 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685674    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:25.685732 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685695    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:25.685732 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685710    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:25.685732 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.685726    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3542c449-0d85-4534-9284-67c7250ccd64-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:25.687183 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.687146    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-istio-token" pod "3542c449-0d85-4534-9284-67c7250ccd64" (UID: "3542c449-0d85-4534-9284-67c7250ccd64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:11:25.687283 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.687195    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-envoy" pod "3542c449-0d85-4534-9284-67c7250ccd64" (UID: "3542c449-0d85-4534-9284-67c7250ccd64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:11:25.687283 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.687231    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3542c449-0d85-4534-9284-67c7250ccd64-istio-podinfo" pod "3542c449-0d85-4534-9284-67c7250ccd64" (UID: "3542c449-0d85-4534-9284-67c7250ccd64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:11:25.687392 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.687370    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-kube-api-access-9n4th" pod "3542c449-0d85-4534-9284-67c7250ccd64" (UID: "3542c449-0d85-4534-9284-67c7250ccd64"). InnerVolumeSpecName "kube-api-access-9n4th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:11:25.786607 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.786570    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:25.786607 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.786601    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3542c449-0d85-4534-9284-67c7250ccd64-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:25.786607 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.786611    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n4th\" (UniqueName: \"kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-kube-api-access-9n4th\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:25.786894 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.786623    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3542c449-0d85-4534-9284-67c7250ccd64-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:25.786894 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:25.786631    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3542c449-0d85-4534-9284-67c7250ccd64-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:26.079225 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:26.079182    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"]
Nov 02 21:11:26.083866 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:26.083828    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-gc9b5"]
Nov 02 21:11:27.167035 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:27.166994    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3542c449-0d85-4534-9284-67c7250ccd64" path="/var/lib/kubelet/pods/3542c449-0d85-4534-9284-67c7250ccd64/volumes"
Nov 02 21:11:27.358758 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:11:27.358720    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:11:27.755728 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:27.755679    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:32.128536 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:32.128500    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:11:32.684107 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:32.684066    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:37.644494 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:37.644456    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:39.343844 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:39.343804    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:42.568228 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:42.568191    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:45.001469 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:45.001436    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:45.196927 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:45.196881    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:47.458903 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:47.458865    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:47.866755 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:47.866714    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:50.331307 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:50.331264    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:52.356695 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:52.356662    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:52.769229 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:52.769196    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:54.259813 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:54.259763    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:54.506280 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:54.506240    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"]
Nov 02 21:11:54.922330 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:54.922293    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:55.095182 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:55.095141    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:57.359668 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:11:57.359619    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:11:57.643344 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:57.643246    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:11:59.506625 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:59.506585    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf" podUID="78e38acc-d4d7-43eb-abc8-323a9723f67e" containerName="payload-processing" containerID="cri-o://d7efb59b0d091a5f57b52b04ff5a41cc090b07f39047913e29335ded2308199e" gracePeriod=25
Nov 02 21:11:59.760852 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:11:59.760749    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:02.412409 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:02.412372    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf" podUID="78e38acc-d4d7-43eb-abc8-323a9723f67e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 21:12:02.534203 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:02.534162    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:04.345588 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:04.345549    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:04.647759 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:04.647678    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:07.505906 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:07.505873    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:12.354616 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:12.354577    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:12.412544 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:12.412508    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf" podUID="78e38acc-d4d7-43eb-abc8-323a9723f67e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 21:12:12.813047 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:12.813009    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:13.532551 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:13.532513    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:17.251829 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:17.251789    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:19.763743 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.763722    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"
Nov 02 21:12:19.843786 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.843739    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e38acc-d4d7-43eb-abc8-323a9723f67e-kube-api-access-48c6q\" (UniqueName: \"kubernetes.io/projected/78e38acc-d4d7-43eb-abc8-323a9723f67e-kube-api-access-48c6q\") pod \"78e38acc-d4d7-43eb-abc8-323a9723f67e\" (UID: \"78e38acc-d4d7-43eb-abc8-323a9723f67e\") "
Nov 02 21:12:19.843967 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.843801    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e38acc-d4d7-43eb-abc8-323a9723f67e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e38acc-d4d7-43eb-abc8-323a9723f67e-plugins-config-volume\") pod \"78e38acc-d4d7-43eb-abc8-323a9723f67e\" (UID: \"78e38acc-d4d7-43eb-abc8-323a9723f67e\") "
Nov 02 21:12:19.844220 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.844188    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e38acc-d4d7-43eb-abc8-323a9723f67e-plugins-config-volume" pod "78e38acc-d4d7-43eb-abc8-323a9723f67e" (UID: "78e38acc-d4d7-43eb-abc8-323a9723f67e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:12:19.845793 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.845774    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e38acc-d4d7-43eb-abc8-323a9723f67e-kube-api-access-48c6q" pod "78e38acc-d4d7-43eb-abc8-323a9723f67e" (UID: "78e38acc-d4d7-43eb-abc8-323a9723f67e"). InnerVolumeSpecName "kube-api-access-48c6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:12:19.944990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.944956    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48c6q\" (UniqueName: \"kubernetes.io/projected/78e38acc-d4d7-43eb-abc8-323a9723f67e-kube-api-access-48c6q\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:19.944990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.944985    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e38acc-d4d7-43eb-abc8-323a9723f67e-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:19.955349 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.955317    5224 generic.go:356] "Generic (PLEG): container finished" podID="78e38acc-d4d7-43eb-abc8-323a9723f67e" containerID="d7efb59b0d091a5f57b52b04ff5a41cc090b07f39047913e29335ded2308199e" exitCode=0
Nov 02 21:12:19.955487 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.955361    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"
Nov 02 21:12:19.955487 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.955405    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf" event={"ID":"78e38acc-d4d7-43eb-abc8-323a9723f67e","Type":"ContainerDied","Data":"d7efb59b0d091a5f57b52b04ff5a41cc090b07f39047913e29335ded2308199e"}
Nov 02 21:12:19.955487 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.955448    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf" event={"ID":"78e38acc-d4d7-43eb-abc8-323a9723f67e","Type":"ContainerDied","Data":"d302a7e5d7d5fd9663dfdbdd48320cc2616aa1c8e7b43af6c669ec4ab5cdb7e9"}
Nov 02 21:12:19.955487 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.955471    5224 scope.go:122] "RemoveContainer" containerID="d7efb59b0d091a5f57b52b04ff5a41cc090b07f39047913e29335ded2308199e"
Nov 02 21:12:19.964836 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.964818    5224 scope.go:122] "RemoveContainer" containerID="d7efb59b0d091a5f57b52b04ff5a41cc090b07f39047913e29335ded2308199e"
Nov 02 21:12:19.965090 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:12:19.965071    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7efb59b0d091a5f57b52b04ff5a41cc090b07f39047913e29335ded2308199e\": container with ID starting with d7efb59b0d091a5f57b52b04ff5a41cc090b07f39047913e29335ded2308199e not found: ID does not exist" containerID="d7efb59b0d091a5f57b52b04ff5a41cc090b07f39047913e29335ded2308199e"
Nov 02 21:12:19.965223 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:19.965103    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7efb59b0d091a5f57b52b04ff5a41cc090b07f39047913e29335ded2308199e"} err="failed to get container status \"d7efb59b0d091a5f57b52b04ff5a41cc090b07f39047913e29335ded2308199e\": rpc error: code = NotFound desc = could not find container \"d7efb59b0d091a5f57b52b04ff5a41cc090b07f39047913e29335ded2308199e\": container with ID starting with d7efb59b0d091a5f57b52b04ff5a41cc090b07f39047913e29335ded2308199e not found: ID does not exist"
Nov 02 21:12:20.120975 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:20.120898    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"]
Nov 02 21:12:20.125762 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:20.125731    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a9ed60-6f487474cf-bb2sf"]
Nov 02 21:12:21.167303 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.167271    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e38acc-d4d7-43eb-abc8-323a9723f67e" path="/var/lib/kubelet/pods/78e38acc-d4d7-43eb-abc8-323a9723f67e/volumes"
Nov 02 21:12:21.319480 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.319443    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"]
Nov 02 21:12:21.319738 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.319711    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj" podUID="bdb0be65-a5fd-4ea9-8573-d9a755151a44" containerName="maas-api" containerID="cri-o://fa47e8e2863cb6ee27e8b40164653a251c91d317f985ab8002ba971884172c30" gracePeriod=30
Nov 02 21:12:21.472155 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.472116    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:21.612978 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.612944    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 21:12:21.663447 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.663410    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb0be65-a5fd-4ea9-8573-d9a755151a44-kube-api-access-72nw7\" (UniqueName: \"kubernetes.io/projected/bdb0be65-a5fd-4ea9-8573-d9a755151a44-kube-api-access-72nw7\") pod \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") "
Nov 02 21:12:21.663646 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.663471    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdb0be65-a5fd-4ea9-8573-d9a755151a44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb0be65-a5fd-4ea9-8573-d9a755151a44-openshift-service-ca\") pod \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") "
Nov 02 21:12:21.663646 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.663594    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-metrics-certs\") pod \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") "
Nov 02 21:12:21.663792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.663740    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-maas-api-tls\") pod \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\" (UID: \"bdb0be65-a5fd-4ea9-8573-d9a755151a44\") "
Nov 02 21:12:21.663869 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.663850    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb0be65-a5fd-4ea9-8573-d9a755151a44-openshift-service-ca" pod "bdb0be65-a5fd-4ea9-8573-d9a755151a44" (UID: "bdb0be65-a5fd-4ea9-8573-d9a755151a44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:12:21.664093 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.664071    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb0be65-a5fd-4ea9-8573-d9a755151a44-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:21.665558 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.665537    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb0be65-a5fd-4ea9-8573-d9a755151a44-kube-api-access-72nw7" pod "bdb0be65-a5fd-4ea9-8573-d9a755151a44" (UID: "bdb0be65-a5fd-4ea9-8573-d9a755151a44"). InnerVolumeSpecName "kube-api-access-72nw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:12:21.666028 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.666009    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-maas-api-tls" pod "bdb0be65-a5fd-4ea9-8573-d9a755151a44" (UID: "bdb0be65-a5fd-4ea9-8573-d9a755151a44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:12:21.666119 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.666030    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-metrics-certs" pod "bdb0be65-a5fd-4ea9-8573-d9a755151a44" (UID: "bdb0be65-a5fd-4ea9-8573-d9a755151a44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:12:21.764746 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.764719    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:21.764746 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.764747    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdb0be65-a5fd-4ea9-8573-d9a755151a44-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:21.764970 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.764758    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72nw7\" (UniqueName: \"kubernetes.io/projected/bdb0be65-a5fd-4ea9-8573-d9a755151a44-kube-api-access-72nw7\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:21.967819 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.967785    5224 generic.go:356] "Generic (PLEG): container finished" podID="bdb0be65-a5fd-4ea9-8573-d9a755151a44" containerID="fa47e8e2863cb6ee27e8b40164653a251c91d317f985ab8002ba971884172c30" exitCode=0
Nov 02 21:12:21.967972 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.967841    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"
Nov 02 21:12:21.967972 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.967856    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj" event={"ID":"bdb0be65-a5fd-4ea9-8573-d9a755151a44","Type":"ContainerDied","Data":"fa47e8e2863cb6ee27e8b40164653a251c91d317f985ab8002ba971884172c30"}
Nov 02 21:12:21.967972 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.967908    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj" event={"ID":"bdb0be65-a5fd-4ea9-8573-d9a755151a44","Type":"ContainerDied","Data":"0f89d36b3934a9acef577fecd7e58356ffe57f6af990888b413e3c8e1f8917d4"}
Nov 02 21:12:21.967972 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.967932    5224 scope.go:122] "RemoveContainer" containerID="fa47e8e2863cb6ee27e8b40164653a251c91d317f985ab8002ba971884172c30"
Nov 02 21:12:21.977787 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.977749    5224 scope.go:122] "RemoveContainer" containerID="fa47e8e2863cb6ee27e8b40164653a251c91d317f985ab8002ba971884172c30"
Nov 02 21:12:21.978063 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:12:21.978043    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa47e8e2863cb6ee27e8b40164653a251c91d317f985ab8002ba971884172c30\": container with ID starting with fa47e8e2863cb6ee27e8b40164653a251c91d317f985ab8002ba971884172c30 not found: ID does not exist" containerID="fa47e8e2863cb6ee27e8b40164653a251c91d317f985ab8002ba971884172c30"
Nov 02 21:12:21.978132 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:21.978070    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa47e8e2863cb6ee27e8b40164653a251c91d317f985ab8002ba971884172c30"} err="failed to get container status \"fa47e8e2863cb6ee27e8b40164653a251c91d317f985ab8002ba971884172c30\": rpc error: code = NotFound desc = could not find container \"fa47e8e2863cb6ee27e8b40164653a251c91d317f985ab8002ba971884172c30\": container with ID starting with fa47e8e2863cb6ee27e8b40164653a251c91d317f985ab8002ba971884172c30 not found: ID does not exist"
Nov 02 21:12:22.134430 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:22.134347    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"]
Nov 02 21:12:22.139168 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:22.139127    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a9ed60-58b7d5ccb9-zzfmj"]
Nov 02 21:12:23.167223 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:23.167191    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdb0be65-a5fd-4ea9-8573-d9a755151a44" path="/var/lib/kubelet/pods/bdb0be65-a5fd-4ea9-8573-d9a755151a44/volumes"
Nov 02 21:12:23.242668 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:23.242632    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:23.473334 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:23.473234    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:24.810105 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:24.810061    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:25.166575 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:25.166500    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:12:25.335610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:25.335572    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:26.114371 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:26.114337    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:27.357743 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:12:27.357703    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:12:28.813300 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:28.813267    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:30.270928 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:30.270899    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:31.680946 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:31.680903    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:31.917208 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:31.917171    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:32.471277 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:32.471231    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:33.993927 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:33.993891    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:38.429728 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:38.429687    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:39.306705 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:39.306670    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:41.617062 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:41.616994    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:42.671574 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:42.671539    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"]
Nov 02 21:12:42.721220 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:42.721185    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:43.590628 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:43.590591    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"]
Nov 02 21:12:43.644499 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:43.644463    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:44.026325 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:44.026291    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:44.210998 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:44.210960    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:47.149578 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:47.149541    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:47.340429 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:47.340394    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:48.856037 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:48.855957    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:49.127576 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:49.127474    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:51.719085 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:51.719052    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:12:53.777739 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:53.777651    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:57.358186 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:12:57.358148    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:12:57.691306 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:57.691217    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4" podUID="de83a78c-767c-4f9f-a0b4-3560413bbba7" containerName="main" containerID="cri-o://71ced0fcda7a7cc896e13706091e2b55d815a95e4e559ecfed8a3b444b3f8dad" gracePeriod=45
Nov 02 21:12:57.948124 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:57.948059    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:12:58.028037 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.028001    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-home\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-home\") pod \"de83a78c-767c-4f9f-a0b4-3560413bbba7\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") "
Nov 02 21:12:58.028224 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.028044    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-model-cache\") pod \"de83a78c-767c-4f9f-a0b4-3560413bbba7\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") "
Nov 02 21:12:58.028224 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.028075    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-dshm\") pod \"de83a78c-767c-4f9f-a0b4-3560413bbba7\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") "
Nov 02 21:12:58.028224 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.028113    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de83a78c-767c-4f9f-a0b4-3560413bbba7-tls-certs\" (UniqueName: \"kubernetes.io/secret/de83a78c-767c-4f9f-a0b4-3560413bbba7-tls-certs\") pod \"de83a78c-767c-4f9f-a0b4-3560413bbba7\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") "
Nov 02 21:12:58.028398 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.028224    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de83a78c-767c-4f9f-a0b4-3560413bbba7-kube-api-access-s88mj\" (UniqueName: \"kubernetes.io/projected/de83a78c-767c-4f9f-a0b4-3560413bbba7-kube-api-access-s88mj\") pod \"de83a78c-767c-4f9f-a0b4-3560413bbba7\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") "
Nov 02 21:12:58.028398 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.028255    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-tmp-dir\") pod \"de83a78c-767c-4f9f-a0b4-3560413bbba7\" (UID: \"de83a78c-767c-4f9f-a0b4-3560413bbba7\") "
Nov 02 21:12:58.028398 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.028286    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-home" pod "de83a78c-767c-4f9f-a0b4-3560413bbba7" (UID: "de83a78c-767c-4f9f-a0b4-3560413bbba7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:12:58.028398 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.028310    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-model-cache" pod "de83a78c-767c-4f9f-a0b4-3560413bbba7" (UID: "de83a78c-767c-4f9f-a0b4-3560413bbba7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:12:58.028616 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.028549    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-tmp-dir" pod "de83a78c-767c-4f9f-a0b4-3560413bbba7" (UID: "de83a78c-767c-4f9f-a0b4-3560413bbba7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:12:58.028658 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.028611    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:58.028658 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.028633    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:58.030363 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.030340    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-dshm" pod "de83a78c-767c-4f9f-a0b4-3560413bbba7" (UID: "de83a78c-767c-4f9f-a0b4-3560413bbba7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:12:58.030461 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.030398    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de83a78c-767c-4f9f-a0b4-3560413bbba7-tls-certs" pod "de83a78c-767c-4f9f-a0b4-3560413bbba7" (UID: "de83a78c-767c-4f9f-a0b4-3560413bbba7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:12:58.030461 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.030443    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de83a78c-767c-4f9f-a0b4-3560413bbba7-kube-api-access-s88mj" pod "de83a78c-767c-4f9f-a0b4-3560413bbba7" (UID: "de83a78c-767c-4f9f-a0b4-3560413bbba7"). InnerVolumeSpecName "kube-api-access-s88mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:12:58.130159 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.130126    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:58.130159 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.130152    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de83a78c-767c-4f9f-a0b4-3560413bbba7-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:58.130159 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.130163    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s88mj\" (UniqueName: \"kubernetes.io/projected/de83a78c-767c-4f9f-a0b4-3560413bbba7-kube-api-access-s88mj\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:58.130395 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.130194    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de83a78c-767c-4f9f-a0b4-3560413bbba7-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:58.161462 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.161426    5224 generic.go:356] "Generic (PLEG): container finished" podID="de83a78c-767c-4f9f-a0b4-3560413bbba7" containerID="71ced0fcda7a7cc896e13706091e2b55d815a95e4e559ecfed8a3b444b3f8dad" exitCode=0
Nov 02 21:12:58.161646 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.161468    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"
Nov 02 21:12:58.161646 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.161504    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4" event={"ID":"de83a78c-767c-4f9f-a0b4-3560413bbba7","Type":"ContainerDied","Data":"71ced0fcda7a7cc896e13706091e2b55d815a95e4e559ecfed8a3b444b3f8dad"}
Nov 02 21:12:58.161646 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.161546    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4" event={"ID":"de83a78c-767c-4f9f-a0b4-3560413bbba7","Type":"ContainerDied","Data":"a12f5ba0bd8e1f7b58f46aba83e27bfdb90960e9262fc81fa06b4ae191a654d7"}
Nov 02 21:12:58.161646 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.161569    5224 scope.go:122] "RemoveContainer" containerID="71ced0fcda7a7cc896e13706091e2b55d815a95e4e559ecfed8a3b444b3f8dad"
Nov 02 21:12:58.170823 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.170799    5224 scope.go:122] "RemoveContainer" containerID="71ced0fcda7a7cc896e13706091e2b55d815a95e4e559ecfed8a3b444b3f8dad"
Nov 02 21:12:58.171102 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:12:58.171083    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ced0fcda7a7cc896e13706091e2b55d815a95e4e559ecfed8a3b444b3f8dad\": container with ID starting with 71ced0fcda7a7cc896e13706091e2b55d815a95e4e559ecfed8a3b444b3f8dad not found: ID does not exist" containerID="71ced0fcda7a7cc896e13706091e2b55d815a95e4e559ecfed8a3b444b3f8dad"
Nov 02 21:12:58.171177 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.171115    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ced0fcda7a7cc896e13706091e2b55d815a95e4e559ecfed8a3b444b3f8dad"} err="failed to get container status \"71ced0fcda7a7cc896e13706091e2b55d815a95e4e559ecfed8a3b444b3f8dad\": rpc error: code = NotFound desc = could not find container \"71ced0fcda7a7cc896e13706091e2b55d815a95e4e559ecfed8a3b444b3f8dad\": container with ID starting with 71ced0fcda7a7cc896e13706091e2b55d815a95e4e559ecfed8a3b444b3f8dad not found: ID does not exist"
Nov 02 21:12:58.327633 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.327599    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"]
Nov 02 21:12:58.332970 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.332945    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-44a484/rate-test-model-44a484-kserve-55fb866d97-m4hm4"]
Nov 02 21:12:58.610738 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.610631    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg" podUID="bc611f03-fc41-41c6-a631-d0b344a7c8f9" containerName="main" containerID="cri-o://9e41f48440d947f2f0513fa7702fd8e8180a0f31f59e5f55bc6f26b9176cc359" gracePeriod=45
Nov 02 21:12:58.664977 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.664935    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:12:58.857854 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.857830    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:12:58.938029 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.937951    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-dshm\") pod \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") "
Nov 02 21:12:58.938029 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.938014    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tls-certs\") pod \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") "
Nov 02 21:12:58.938259 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.938050    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tmp-dir\") pod \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") "
Nov 02 21:12:58.938259 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.938076    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-home\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-home\") pod \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") "
Nov 02 21:12:58.938259 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.938120    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-model-cache\") pod \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") "
Nov 02 21:12:58.938259 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.938166    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc611f03-fc41-41c6-a631-d0b344a7c8f9-kube-api-access-zpbw6\" (UniqueName: \"kubernetes.io/projected/bc611f03-fc41-41c6-a631-d0b344a7c8f9-kube-api-access-zpbw6\") pod \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\" (UID: \"bc611f03-fc41-41c6-a631-d0b344a7c8f9\") "
Nov 02 21:12:58.938492 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.938343    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tmp-dir" pod "bc611f03-fc41-41c6-a631-d0b344a7c8f9" (UID: "bc611f03-fc41-41c6-a631-d0b344a7c8f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:12:58.938492 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.938362    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-home" pod "bc611f03-fc41-41c6-a631-d0b344a7c8f9" (UID: "bc611f03-fc41-41c6-a631-d0b344a7c8f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:12:58.938492 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.938431    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-model-cache" pod "bc611f03-fc41-41c6-a631-d0b344a7c8f9" (UID: "bc611f03-fc41-41c6-a631-d0b344a7c8f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:12:58.940043 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.940014    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-dshm" pod "bc611f03-fc41-41c6-a631-d0b344a7c8f9" (UID: "bc611f03-fc41-41c6-a631-d0b344a7c8f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:12:58.940173 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.940121    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tls-certs" pod "bc611f03-fc41-41c6-a631-d0b344a7c8f9" (UID: "bc611f03-fc41-41c6-a631-d0b344a7c8f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:12:58.940335 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:58.940301    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc611f03-fc41-41c6-a631-d0b344a7c8f9-kube-api-access-zpbw6" pod "bc611f03-fc41-41c6-a631-d0b344a7c8f9" (UID: "bc611f03-fc41-41c6-a631-d0b344a7c8f9"). InnerVolumeSpecName "kube-api-access-zpbw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:12:59.039473 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.039445    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:59.039473 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.039471    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:59.039674 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.039481    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:59.039674 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.039491    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:59.039674 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.039500    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc611f03-fc41-41c6-a631-d0b344a7c8f9-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:59.039674 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.039510    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpbw6\" (UniqueName: \"kubernetes.io/projected/bc611f03-fc41-41c6-a631-d0b344a7c8f9-kube-api-access-zpbw6\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:12:59.166442 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.166408    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de83a78c-767c-4f9f-a0b4-3560413bbba7" path="/var/lib/kubelet/pods/de83a78c-767c-4f9f-a0b4-3560413bbba7/volumes"
Nov 02 21:12:59.170239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.170208    5224 generic.go:356] "Generic (PLEG): container finished" podID="bc611f03-fc41-41c6-a631-d0b344a7c8f9" containerID="9e41f48440d947f2f0513fa7702fd8e8180a0f31f59e5f55bc6f26b9176cc359" exitCode=0
Nov 02 21:12:59.170395 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.170248    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"
Nov 02 21:12:59.170395 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.170250    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg" event={"ID":"bc611f03-fc41-41c6-a631-d0b344a7c8f9","Type":"ContainerDied","Data":"9e41f48440d947f2f0513fa7702fd8e8180a0f31f59e5f55bc6f26b9176cc359"}
Nov 02 21:12:59.170395 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.170365    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg" event={"ID":"bc611f03-fc41-41c6-a631-d0b344a7c8f9","Type":"ContainerDied","Data":"eeef76102c0b414b20572397d70f8fe7f77f0bfd53fe16e477f5fd8a2a975073"}
Nov 02 21:12:59.170395 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.170384    5224 scope.go:122] "RemoveContainer" containerID="9e41f48440d947f2f0513fa7702fd8e8180a0f31f59e5f55bc6f26b9176cc359"
Nov 02 21:12:59.181736 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.181717    5224 scope.go:122] "RemoveContainer" containerID="9e41f48440d947f2f0513fa7702fd8e8180a0f31f59e5f55bc6f26b9176cc359"
Nov 02 21:12:59.182008 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:12:59.181987    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e41f48440d947f2f0513fa7702fd8e8180a0f31f59e5f55bc6f26b9176cc359\": container with ID starting with 9e41f48440d947f2f0513fa7702fd8e8180a0f31f59e5f55bc6f26b9176cc359 not found: ID does not exist" containerID="9e41f48440d947f2f0513fa7702fd8e8180a0f31f59e5f55bc6f26b9176cc359"
Nov 02 21:12:59.182077 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.182016    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e41f48440d947f2f0513fa7702fd8e8180a0f31f59e5f55bc6f26b9176cc359"} err="failed to get container status \"9e41f48440d947f2f0513fa7702fd8e8180a0f31f59e5f55bc6f26b9176cc359\": rpc error: code = NotFound desc = could not find container \"9e41f48440d947f2f0513fa7702fd8e8180a0f31f59e5f55bc6f26b9176cc359\": container with ID starting with 9e41f48440d947f2f0513fa7702fd8e8180a0f31f59e5f55bc6f26b9176cc359 not found: ID does not exist"
Nov 02 21:12:59.336737 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.336702    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"]
Nov 02 21:12:59.339316 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:12:59.339294    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-09ecd0/rate-test-model-09ecd0-kserve-cb65595c-55zzg"]
Nov 02 21:13:01.167035 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:01.167002    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc611f03-fc41-41c6-a631-d0b344a7c8f9" path="/var/lib/kubelet/pods/bc611f03-fc41-41c6-a631-d0b344a7c8f9/volumes"
Nov 02 21:13:03.593235 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:03.593200    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:07.609311 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:07.609274    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"]
Nov 02 21:13:07.609741 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:07.609489    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj" podUID="f93a40bb-5ce3-4355-b2d4-6228d7857424" containerName="maas-api" containerID="cri-o://c955b57dd6ced0ab67ab9a98bf3064a717c825e0ba4b5bd5dfdd4f89d90595d1" gracePeriod=30
Nov 02 21:13:07.757716 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:07.757662    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:07.862617 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:07.862544    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 21:13:07.928951 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:07.928910    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93a40bb-5ce3-4355-b2d4-6228d7857424-kube-api-access-ztswm\" (UniqueName: \"kubernetes.io/projected/f93a40bb-5ce3-4355-b2d4-6228d7857424-kube-api-access-ztswm\") pod \"f93a40bb-5ce3-4355-b2d4-6228d7857424\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") "
Nov 02 21:13:07.928951 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:07.928951    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f93a40bb-5ce3-4355-b2d4-6228d7857424-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93a40bb-5ce3-4355-b2d4-6228d7857424-openshift-service-ca\") pod \"f93a40bb-5ce3-4355-b2d4-6228d7857424\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") "
Nov 02 21:13:07.929214 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:07.928994    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-maas-api-tls\") pod \"f93a40bb-5ce3-4355-b2d4-6228d7857424\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") "
Nov 02 21:13:07.929214 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:07.929021    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-metrics-certs\") pod \"f93a40bb-5ce3-4355-b2d4-6228d7857424\" (UID: \"f93a40bb-5ce3-4355-b2d4-6228d7857424\") "
Nov 02 21:13:07.929396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:07.929360    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93a40bb-5ce3-4355-b2d4-6228d7857424-openshift-service-ca" pod "f93a40bb-5ce3-4355-b2d4-6228d7857424" (UID: "f93a40bb-5ce3-4355-b2d4-6228d7857424"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:13:07.931188 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:07.931156    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-metrics-certs" pod "f93a40bb-5ce3-4355-b2d4-6228d7857424" (UID: "f93a40bb-5ce3-4355-b2d4-6228d7857424"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:13:07.931605 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:07.931580    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-maas-api-tls" pod "f93a40bb-5ce3-4355-b2d4-6228d7857424" (UID: "f93a40bb-5ce3-4355-b2d4-6228d7857424"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:13:07.931605 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:07.931597    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93a40bb-5ce3-4355-b2d4-6228d7857424-kube-api-access-ztswm" pod "f93a40bb-5ce3-4355-b2d4-6228d7857424" (UID: "f93a40bb-5ce3-4355-b2d4-6228d7857424"). InnerVolumeSpecName "kube-api-access-ztswm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:13:08.030846 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:08.030815    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:08.030846 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:08.030844    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f93a40bb-5ce3-4355-b2d4-6228d7857424-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:08.030846 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:08.030854    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztswm\" (UniqueName: \"kubernetes.io/projected/f93a40bb-5ce3-4355-b2d4-6228d7857424-kube-api-access-ztswm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:08.031092 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:08.030865    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93a40bb-5ce3-4355-b2d4-6228d7857424-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:08.217314 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:08.217230    5224 generic.go:356] "Generic (PLEG): container finished" podID="f93a40bb-5ce3-4355-b2d4-6228d7857424" containerID="c955b57dd6ced0ab67ab9a98bf3064a717c825e0ba4b5bd5dfdd4f89d90595d1" exitCode=0
Nov 02 21:13:08.217314 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:08.217281    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"
Nov 02 21:13:08.217526 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:08.217309    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj" event={"ID":"f93a40bb-5ce3-4355-b2d4-6228d7857424","Type":"ContainerDied","Data":"c955b57dd6ced0ab67ab9a98bf3064a717c825e0ba4b5bd5dfdd4f89d90595d1"}
Nov 02 21:13:08.217526 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:08.217363    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj" event={"ID":"f93a40bb-5ce3-4355-b2d4-6228d7857424","Type":"ContainerDied","Data":"e2c1afb961756e83d36dfce93db690353a6b62fda4a5640b92710cf1c8f878bf"}
Nov 02 21:13:08.217526 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:08.217385    5224 scope.go:122] "RemoveContainer" containerID="c955b57dd6ced0ab67ab9a98bf3064a717c825e0ba4b5bd5dfdd4f89d90595d1"
Nov 02 21:13:08.233618 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:08.233592    5224 scope.go:122] "RemoveContainer" containerID="c955b57dd6ced0ab67ab9a98bf3064a717c825e0ba4b5bd5dfdd4f89d90595d1"
Nov 02 21:13:08.233969 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:13:08.233935    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c955b57dd6ced0ab67ab9a98bf3064a717c825e0ba4b5bd5dfdd4f89d90595d1\": container with ID starting with c955b57dd6ced0ab67ab9a98bf3064a717c825e0ba4b5bd5dfdd4f89d90595d1 not found: ID does not exist" containerID="c955b57dd6ced0ab67ab9a98bf3064a717c825e0ba4b5bd5dfdd4f89d90595d1"
Nov 02 21:13:08.234117 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:08.233979    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c955b57dd6ced0ab67ab9a98bf3064a717c825e0ba4b5bd5dfdd4f89d90595d1"} err="failed to get container status \"c955b57dd6ced0ab67ab9a98bf3064a717c825e0ba4b5bd5dfdd4f89d90595d1\": rpc error: code = NotFound desc = could not find container \"c955b57dd6ced0ab67ab9a98bf3064a717c825e0ba4b5bd5dfdd4f89d90595d1\": container with ID starting with c955b57dd6ced0ab67ab9a98bf3064a717c825e0ba4b5bd5dfdd4f89d90595d1 not found: ID does not exist"
Nov 02 21:13:08.382954 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:08.382921    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"]
Nov 02 21:13:08.388461 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:08.388440    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj"]
Nov 02 21:13:09.166115 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:09.166080    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93a40bb-5ce3-4355-b2d4-6228d7857424" path="/var/lib/kubelet/pods/f93a40bb-5ce3-4355-b2d4-6228d7857424/volumes"
Nov 02 21:13:09.621902 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:09.621849    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:10.232862 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:10.232806    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:10.418840 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:10.418806    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:10.787663 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:10.787625    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-578636-5b884999bb-grlbj" podUID="f93a40bb-5ce3-4355-b2d4-6228d7857424" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.147:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:13:11.768999 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:11.768962    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:12.294289 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:12.294260    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:12.754010 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:12.753924    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:13.144969 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:13.144934    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:13.858541 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:13.858506    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:14.905948 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:14.905897    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:16.954173 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:16.954135    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:17.154758 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:17.154719    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:17.667666 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:17.667634    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"]
Nov 02 21:13:17.719358 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:17.719325    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"]
Nov 02 21:13:17.858124 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:17.858088    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:18.132929 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:18.132888    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:18.311779 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:18.311734    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:18.612220 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:18.612187    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:18.790071 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:18.790041    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:19.166535 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:19.166506    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:13:20.107038 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:20.106997    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:23.974671 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:23.974626    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:25.795963 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:25.795926    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:26.365490 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:26.365454    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:27.358619 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:13:27.358585    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:13:30.022707 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:30.022663    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:30.328205 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:30.328115    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:30.996325 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:30.996276    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:31.219003 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:31.218970    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:32.687273 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:32.687227    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh" podUID="3359dc52-961f-43fd-b567-3bcb22271302" containerName="main" containerID="cri-o://6c35ac8eda2718c2774bd2ef412b1abc6410f3c821a53ebfe6bf8ae0cb5abc23" gracePeriod=45
Nov 02 21:13:32.741230 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:32.741187    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj" podUID="f4ff50fb-2482-4330-9ce4-e05d794c2a2a" containerName="main" containerID="cri-o://181de45ea58399596f802acae56c7ef80189d2f6249bb9bf00be5388ba9ffabc" gracePeriod=45
Nov 02 21:13:33.003297 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.003272    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 21:13:33.006102 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.006083    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 21:13:33.076193 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076163    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-dshm\") pod \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") "
Nov 02 21:13:33.076369 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076213    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-home\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-home\") pod \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") "
Nov 02 21:13:33.076369 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076258    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-tmp-dir\") pod \"3359dc52-961f-43fd-b567-3bcb22271302\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") "
Nov 02 21:13:33.076369 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076279    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tls-certs\") pod \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") "
Nov 02 21:13:33.076369 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076302    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-model-cache\") pod \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") "
Nov 02 21:13:33.076369 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076325    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3359dc52-961f-43fd-b567-3bcb22271302-tls-certs\" (UniqueName: \"kubernetes.io/secret/3359dc52-961f-43fd-b567-3bcb22271302-tls-certs\") pod \"3359dc52-961f-43fd-b567-3bcb22271302\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") "
Nov 02 21:13:33.076369 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076355    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3359dc52-961f-43fd-b567-3bcb22271302-kube-api-access-rwm8b\" (UniqueName: \"kubernetes.io/projected/3359dc52-961f-43fd-b567-3bcb22271302-kube-api-access-rwm8b\") pod \"3359dc52-961f-43fd-b567-3bcb22271302\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") "
Nov 02 21:13:33.076703 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076389    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-dshm\") pod \"3359dc52-961f-43fd-b567-3bcb22271302\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") "
Nov 02 21:13:33.076703 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076432    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-model-cache\") pod \"3359dc52-961f-43fd-b567-3bcb22271302\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") "
Nov 02 21:13:33.076703 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076488    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-home\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-home\") pod \"3359dc52-961f-43fd-b567-3bcb22271302\" (UID: \"3359dc52-961f-43fd-b567-3bcb22271302\") "
Nov 02 21:13:33.076703 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076519    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tmp-dir\") pod \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") "
Nov 02 21:13:33.076703 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076526    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-tmp-dir" pod "3359dc52-961f-43fd-b567-3bcb22271302" (UID: "3359dc52-961f-43fd-b567-3bcb22271302"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:13:33.076703 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076548    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-kube-api-access-nmn27\" (UniqueName: \"kubernetes.io/projected/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-kube-api-access-nmn27\") pod \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\" (UID: \"f4ff50fb-2482-4330-9ce4-e05d794c2a2a\") "
Nov 02 21:13:33.076703 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076616    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-home" pod "f4ff50fb-2482-4330-9ce4-e05d794c2a2a" (UID: "f4ff50fb-2482-4330-9ce4-e05d794c2a2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:13:33.077118 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076743    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-model-cache" pod "f4ff50fb-2482-4330-9ce4-e05d794c2a2a" (UID: "f4ff50fb-2482-4330-9ce4-e05d794c2a2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:13:33.077118 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.076753    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-home" pod "3359dc52-961f-43fd-b567-3bcb22271302" (UID: "3359dc52-961f-43fd-b567-3bcb22271302"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:13:33.077118 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.077011    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-model-cache" pod "3359dc52-961f-43fd-b567-3bcb22271302" (UID: "3359dc52-961f-43fd-b567-3bcb22271302"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:13:33.077306 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.077187    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:33.077306 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.077206    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:33.077306 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.077220    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:33.077306 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.077220    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tmp-dir" pod "f4ff50fb-2482-4330-9ce4-e05d794c2a2a" (UID: "f4ff50fb-2482-4330-9ce4-e05d794c2a2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:13:33.077306 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.077236    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:33.077306 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.077250    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:33.078699 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.078673    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-dshm" pod "f4ff50fb-2482-4330-9ce4-e05d794c2a2a" (UID: "f4ff50fb-2482-4330-9ce4-e05d794c2a2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:13:33.078898 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.078880    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tls-certs" pod "f4ff50fb-2482-4330-9ce4-e05d794c2a2a" (UID: "f4ff50fb-2482-4330-9ce4-e05d794c2a2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:13:33.079083 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.079053    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3359dc52-961f-43fd-b567-3bcb22271302-tls-certs" pod "3359dc52-961f-43fd-b567-3bcb22271302" (UID: "3359dc52-961f-43fd-b567-3bcb22271302"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:13:33.079083 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.079061    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-dshm" pod "3359dc52-961f-43fd-b567-3bcb22271302" (UID: "3359dc52-961f-43fd-b567-3bcb22271302"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:13:33.079240 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.079129    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3359dc52-961f-43fd-b567-3bcb22271302-kube-api-access-rwm8b" pod "3359dc52-961f-43fd-b567-3bcb22271302" (UID: "3359dc52-961f-43fd-b567-3bcb22271302"). InnerVolumeSpecName "kube-api-access-rwm8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:13:33.079240 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.079148    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-kube-api-access-nmn27" pod "f4ff50fb-2482-4330-9ce4-e05d794c2a2a" (UID: "f4ff50fb-2482-4330-9ce4-e05d794c2a2a"). InnerVolumeSpecName "kube-api-access-nmn27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:13:33.177681 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.177647    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3359dc52-961f-43fd-b567-3bcb22271302-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:33.177681 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.177683    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwm8b\" (UniqueName: \"kubernetes.io/projected/3359dc52-961f-43fd-b567-3bcb22271302-kube-api-access-rwm8b\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:33.178031 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.177699    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3359dc52-961f-43fd-b567-3bcb22271302-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:33.178031 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.177713    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:33.178031 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.177725    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmn27\" (UniqueName: \"kubernetes.io/projected/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-kube-api-access-nmn27\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:33.178031 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.177738    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:33.178031 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.177751    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ff50fb-2482-4330-9ce4-e05d794c2a2a-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:33.342479 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.342447    5224 generic.go:356] "Generic (PLEG): container finished" podID="3359dc52-961f-43fd-b567-3bcb22271302" containerID="6c35ac8eda2718c2774bd2ef412b1abc6410f3c821a53ebfe6bf8ae0cb5abc23" exitCode=0
Nov 02 21:13:33.342661 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.342504    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"
Nov 02 21:13:33.342661 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.342534    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh" event={"ID":"3359dc52-961f-43fd-b567-3bcb22271302","Type":"ContainerDied","Data":"6c35ac8eda2718c2774bd2ef412b1abc6410f3c821a53ebfe6bf8ae0cb5abc23"}
Nov 02 21:13:33.342661 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.342588    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh" event={"ID":"3359dc52-961f-43fd-b567-3bcb22271302","Type":"ContainerDied","Data":"e62c2f6ec156d24d086d5090c51b2fbd95872b3a00a290c339a7f424cd2cf125"}
Nov 02 21:13:33.342661 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.342605    5224 scope.go:122] "RemoveContainer" containerID="6c35ac8eda2718c2774bd2ef412b1abc6410f3c821a53ebfe6bf8ae0cb5abc23"
Nov 02 21:13:33.344235 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.344212    5224 generic.go:356] "Generic (PLEG): container finished" podID="f4ff50fb-2482-4330-9ce4-e05d794c2a2a" containerID="181de45ea58399596f802acae56c7ef80189d2f6249bb9bf00be5388ba9ffabc" exitCode=0
Nov 02 21:13:33.344367 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.344270    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj" event={"ID":"f4ff50fb-2482-4330-9ce4-e05d794c2a2a","Type":"ContainerDied","Data":"181de45ea58399596f802acae56c7ef80189d2f6249bb9bf00be5388ba9ffabc"}
Nov 02 21:13:33.344367 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.344304    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj" event={"ID":"f4ff50fb-2482-4330-9ce4-e05d794c2a2a","Type":"ContainerDied","Data":"356839997d8b307a616aed23b10a5def9ec333e0c8332e59efdca22e0338ed55"}
Nov 02 21:13:33.344367 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.344276    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"
Nov 02 21:13:33.351722 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.351705    5224 scope.go:122] "RemoveContainer" containerID="6c35ac8eda2718c2774bd2ef412b1abc6410f3c821a53ebfe6bf8ae0cb5abc23"
Nov 02 21:13:33.351992 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:13:33.351974    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c35ac8eda2718c2774bd2ef412b1abc6410f3c821a53ebfe6bf8ae0cb5abc23\": container with ID starting with 6c35ac8eda2718c2774bd2ef412b1abc6410f3c821a53ebfe6bf8ae0cb5abc23 not found: ID does not exist" containerID="6c35ac8eda2718c2774bd2ef412b1abc6410f3c821a53ebfe6bf8ae0cb5abc23"
Nov 02 21:13:33.352068 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.352001    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c35ac8eda2718c2774bd2ef412b1abc6410f3c821a53ebfe6bf8ae0cb5abc23"} err="failed to get container status \"6c35ac8eda2718c2774bd2ef412b1abc6410f3c821a53ebfe6bf8ae0cb5abc23\": rpc error: code = NotFound desc = could not find container \"6c35ac8eda2718c2774bd2ef412b1abc6410f3c821a53ebfe6bf8ae0cb5abc23\": container with ID starting with 6c35ac8eda2718c2774bd2ef412b1abc6410f3c821a53ebfe6bf8ae0cb5abc23 not found: ID does not exist"
Nov 02 21:13:33.352068 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.352023    5224 scope.go:122] "RemoveContainer" containerID="181de45ea58399596f802acae56c7ef80189d2f6249bb9bf00be5388ba9ffabc"
Nov 02 21:13:33.360194 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.360168    5224 scope.go:122] "RemoveContainer" containerID="181de45ea58399596f802acae56c7ef80189d2f6249bb9bf00be5388ba9ffabc"
Nov 02 21:13:33.360641 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:13:33.360573    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"181de45ea58399596f802acae56c7ef80189d2f6249bb9bf00be5388ba9ffabc\": container with ID starting with 181de45ea58399596f802acae56c7ef80189d2f6249bb9bf00be5388ba9ffabc not found: ID does not exist" containerID="181de45ea58399596f802acae56c7ef80189d2f6249bb9bf00be5388ba9ffabc"
Nov 02 21:13:33.363945 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.360608    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"181de45ea58399596f802acae56c7ef80189d2f6249bb9bf00be5388ba9ffabc"} err="failed to get container status \"181de45ea58399596f802acae56c7ef80189d2f6249bb9bf00be5388ba9ffabc\": rpc error: code = NotFound desc = could not find container \"181de45ea58399596f802acae56c7ef80189d2f6249bb9bf00be5388ba9ffabc\": container with ID starting with 181de45ea58399596f802acae56c7ef80189d2f6249bb9bf00be5388ba9ffabc not found: ID does not exist"
Nov 02 21:13:33.508391 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.508357    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"]
Nov 02 21:13:33.512472 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.512448    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/distinct-2-w4-c25349-kserve-548f4b7884-qkszh"]
Nov 02 21:13:33.623343 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.623263    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"]
Nov 02 21:13:33.626731 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:33.626707    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/simulator-w4-c25349-kserve-5dbc9cfffc-bhptj"]
Nov 02 21:13:35.167342 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:35.167306    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3359dc52-961f-43fd-b567-3bcb22271302" path="/var/lib/kubelet/pods/3359dc52-961f-43fd-b567-3bcb22271302/volumes"
Nov 02 21:13:35.167789 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:35.167757    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ff50fb-2482-4330-9ce4-e05d794c2a2a" path="/var/lib/kubelet/pods/f4ff50fb-2482-4330-9ce4-e05d794c2a2a/volumes"
Nov 02 21:13:35.266476 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:35.266444    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:36.016745 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:36.016713    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:40.152525 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:40.152486    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:41.670219 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:41.670178    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:43.441499 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:43.441453    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:13:44.982230 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:44.982187    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:49.211589 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:49.211553    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:50.234941 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.234894    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"]
Nov 02 21:13:50.235461 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.235093    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c" podUID="7689b623-5d86-473a-82aa-0739d899ff06" containerName="maas-api" containerID="cri-o://73860c8070484b0a9d8a90b3ddb3dbd46008e800a221797368d6f716035638e0" gracePeriod=30
Nov 02 21:13:50.383468 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.379962    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:50.441827 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.441788    5224 generic.go:356] "Generic (PLEG): container finished" podID="7689b623-5d86-473a-82aa-0739d899ff06" containerID="73860c8070484b0a9d8a90b3ddb3dbd46008e800a221797368d6f716035638e0" exitCode=0
Nov 02 21:13:50.442024 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.441918    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c" event={"ID":"7689b623-5d86-473a-82aa-0739d899ff06","Type":"ContainerDied","Data":"73860c8070484b0a9d8a90b3ddb3dbd46008e800a221797368d6f716035638e0"}
Nov 02 21:13:50.504283 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.504209    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 21:13:50.543732 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.543702    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7689b623-5d86-473a-82aa-0739d899ff06-kube-api-access-wjkw7\" (UniqueName: \"kubernetes.io/projected/7689b623-5d86-473a-82aa-0739d899ff06-kube-api-access-wjkw7\") pod \"7689b623-5d86-473a-82aa-0739d899ff06\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") "
Nov 02 21:13:50.543930 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.543801    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-maas-api-tls\") pod \"7689b623-5d86-473a-82aa-0739d899ff06\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") "
Nov 02 21:13:50.543930 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.543923    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-metrics-certs\") pod \"7689b623-5d86-473a-82aa-0739d899ff06\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") "
Nov 02 21:13:50.544062 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.543955    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7689b623-5d86-473a-82aa-0739d899ff06-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7689b623-5d86-473a-82aa-0739d899ff06-openshift-service-ca\") pod \"7689b623-5d86-473a-82aa-0739d899ff06\" (UID: \"7689b623-5d86-473a-82aa-0739d899ff06\") "
Nov 02 21:13:50.544419 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.544380    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7689b623-5d86-473a-82aa-0739d899ff06-openshift-service-ca" pod "7689b623-5d86-473a-82aa-0739d899ff06" (UID: "7689b623-5d86-473a-82aa-0739d899ff06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:13:50.545966 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.545938    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7689b623-5d86-473a-82aa-0739d899ff06-kube-api-access-wjkw7" pod "7689b623-5d86-473a-82aa-0739d899ff06" (UID: "7689b623-5d86-473a-82aa-0739d899ff06"). InnerVolumeSpecName "kube-api-access-wjkw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:13:50.546080 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.546033    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-maas-api-tls" pod "7689b623-5d86-473a-82aa-0739d899ff06" (UID: "7689b623-5d86-473a-82aa-0739d899ff06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:13:50.546248 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.546229    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-metrics-certs" pod "7689b623-5d86-473a-82aa-0739d899ff06" (UID: "7689b623-5d86-473a-82aa-0739d899ff06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:13:50.645786 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.645729    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:50.645786 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.645785    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7689b623-5d86-473a-82aa-0739d899ff06-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:50.645786 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.645802    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjkw7\" (UniqueName: \"kubernetes.io/projected/7689b623-5d86-473a-82aa-0739d899ff06-kube-api-access-wjkw7\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:50.646138 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:50.645811    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7689b623-5d86-473a-82aa-0739d899ff06-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:13:51.447649 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:51.447568    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c" event={"ID":"7689b623-5d86-473a-82aa-0739d899ff06","Type":"ContainerDied","Data":"43c51ba1fb19fc45732491fdf7f4086ebc650353081cc8af2c6dd3307a3d5aab"}
Nov 02 21:13:51.447649 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:51.447600    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"
Nov 02 21:13:51.447649 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:51.447631    5224 scope.go:122] "RemoveContainer" containerID="73860c8070484b0a9d8a90b3ddb3dbd46008e800a221797368d6f716035638e0"
Nov 02 21:13:51.614303 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:51.614271    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"]
Nov 02 21:13:51.665830 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:51.665798    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c"]
Nov 02 21:13:52.161791 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:52.161742    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:53.166589 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:53.166552    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7689b623-5d86-473a-82aa-0739d899ff06" path="/var/lib/kubelet/pods/7689b623-5d86-473a-82aa-0739d899ff06/volumes"
Nov 02 21:13:53.495524 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:53.495433    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-44a484-6d9b455b64-6nb8c" podUID="7689b623-5d86-473a-82aa-0739d899ff06" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:13:54.624267 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:54.624235    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:54.852900 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:54.852860    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:55.235692 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:55.235660    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:56.170822 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:56.170786    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:56.438165 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:56.438065    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:56.637873 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:56.637837    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:57.104680 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:57.104642    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:57.358057 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:13:57.357979    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:13:57.787266 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:57.787218    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:58.260480 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:58.260403    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:59.244637 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:59.244599    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:13:59.741897 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:13:59.741858    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:00.498503 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:00.498469    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:01.094370 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:01.094325    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:01.276388 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:01.276350    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:01.731644 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:01.731600    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:02.451306 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:02.451271    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:06.554424 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:06.554390    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:06.799354 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:06.799317    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:06.981520 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:06.981433    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:07.591285 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:07.591248    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:09.862811 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:09.862761    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:10.036634 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:10.036597    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:12.330391 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:12.330349    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:12.660062 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:12.659974    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:13.476874 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:13.476840    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:14.434542 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:14.434498    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:14.789937 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:14.789901    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:14.963870 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:14.963823    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:16.557859 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:16.557823    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:14:17.604330 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:17.604296    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:19.096052 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:19.095945    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:19.380543 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:19.380454    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:22.463283 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:22.463234    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:26.671800 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:26.671751    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:27.358495 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:14:27.358464    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:14:28.458501 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:28.458468    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:33.765030 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:33.764982    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:35.019420 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:35.019384    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:35.199594 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:35.199547    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:36.161123 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:36.161091    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:14:36.583609 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:36.583564    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:37.017028 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:37.016993    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"]
Nov 02 21:14:37.017238 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:37.017217    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr" podUID="0eab4cf3-570b-4827-b25d-c01a5683a048" containerName="istio-proxy" containerID="cri-o://7d80df9b2ec74d6f25a367c6b48dd84a284c5acc7bbc90286fb9c7ef32d3b0c5" gracePeriod=30
Nov 02 21:14:37.059013 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:37.058970    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:37.512617 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:37.512583    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:38.920034 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:38.919996    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:41.605513 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:41.605474    5224 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:14:41.605923 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:41.605533    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr" podUID="0eab4cf3-570b-4827-b25d-c01a5683a048" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:14:42.263006 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.262980    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 21:14:42.328266 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328204    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-podinfo\") pod \"0eab4cf3-570b-4827-b25d-c01a5683a048\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") "
Nov 02 21:14:42.328266 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328239    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-certs\") pod \"0eab4cf3-570b-4827-b25d-c01a5683a048\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") "
Nov 02 21:14:42.328266 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328260    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-kube-api-access-k92f8\" (UniqueName: \"kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-kube-api-access-k92f8\") pod \"0eab4cf3-570b-4827-b25d-c01a5683a048\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") "
Nov 02 21:14:42.328537 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328302    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-credential-socket\") pod \"0eab4cf3-570b-4827-b25d-c01a5683a048\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") "
Nov 02 21:14:42.328537 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328347    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eab4cf3-570b-4827-b25d-c01a5683a048-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eab4cf3-570b-4827-b25d-c01a5683a048-istiod-ca-cert\") pod \"0eab4cf3-570b-4827-b25d-c01a5683a048\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") "
Nov 02 21:14:42.328537 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328384    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-socket\") pod \"0eab4cf3-570b-4827-b25d-c01a5683a048\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") "
Nov 02 21:14:42.328537 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328423    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-data\") pod \"0eab4cf3-570b-4827-b25d-c01a5683a048\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") "
Nov 02 21:14:42.328537 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328500    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-envoy\") pod \"0eab4cf3-570b-4827-b25d-c01a5683a048\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") "
Nov 02 21:14:42.328537 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328535    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-token\" (UniqueName: \"kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-token\") pod \"0eab4cf3-570b-4827-b25d-c01a5683a048\" (UID: \"0eab4cf3-570b-4827-b25d-c01a5683a048\") "
Nov 02 21:14:42.328904 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328596    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-certs" pod "0eab4cf3-570b-4827-b25d-c01a5683a048" (UID: "0eab4cf3-570b-4827-b25d-c01a5683a048"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:14:42.328904 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328661    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-credential-socket" pod "0eab4cf3-570b-4827-b25d-c01a5683a048" (UID: "0eab4cf3-570b-4827-b25d-c01a5683a048"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:14:42.328904 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328826    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-data" pod "0eab4cf3-570b-4827-b25d-c01a5683a048" (UID: "0eab4cf3-570b-4827-b25d-c01a5683a048"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:14:42.328904 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328837    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-socket" pod "0eab4cf3-570b-4827-b25d-c01a5683a048" (UID: "0eab4cf3-570b-4827-b25d-c01a5683a048"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:14:42.329134 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328943    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:42.329134 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328962    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:42.329134 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.328976    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:42.329134 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.329014    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:42.329370 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.329130    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eab4cf3-570b-4827-b25d-c01a5683a048-istiod-ca-cert" pod "0eab4cf3-570b-4827-b25d-c01a5683a048" (UID: "0eab4cf3-570b-4827-b25d-c01a5683a048"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:14:42.330437 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.330409    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-kube-api-access-k92f8" pod "0eab4cf3-570b-4827-b25d-c01a5683a048" (UID: "0eab4cf3-570b-4827-b25d-c01a5683a048"). InnerVolumeSpecName "kube-api-access-k92f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:14:42.330556 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.330533    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-envoy" pod "0eab4cf3-570b-4827-b25d-c01a5683a048" (UID: "0eab4cf3-570b-4827-b25d-c01a5683a048"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:14:42.330610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.330586    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-podinfo" pod "0eab4cf3-570b-4827-b25d-c01a5683a048" (UID: "0eab4cf3-570b-4827-b25d-c01a5683a048"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:14:42.330706 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.330690    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-token" pod "0eab4cf3-570b-4827-b25d-c01a5683a048" (UID: "0eab4cf3-570b-4827-b25d-c01a5683a048"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:14:42.429812 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.429782    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:42.429812 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.429810    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:42.430012 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.429820    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eab4cf3-570b-4827-b25d-c01a5683a048-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:42.430012 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.429830    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k92f8\" (UniqueName: \"kubernetes.io/projected/0eab4cf3-570b-4827-b25d-c01a5683a048-kube-api-access-k92f8\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:42.430012 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.429842    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eab4cf3-570b-4827-b25d-c01a5683a048-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:42.486128 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.486097    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"]
Nov 02 21:14:42.618894 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.618812    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:42.692150 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.692116    5224 generic.go:356] "Generic (PLEG): container finished" podID="0eab4cf3-570b-4827-b25d-c01a5683a048" containerID="7d80df9b2ec74d6f25a367c6b48dd84a284c5acc7bbc90286fb9c7ef32d3b0c5" exitCode=0
Nov 02 21:14:42.692319 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.692172    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"
Nov 02 21:14:42.692319 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.692203    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr" event={"ID":"0eab4cf3-570b-4827-b25d-c01a5683a048","Type":"ContainerDied","Data":"7d80df9b2ec74d6f25a367c6b48dd84a284c5acc7bbc90286fb9c7ef32d3b0c5"}
Nov 02 21:14:42.692319 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.692256    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr" event={"ID":"0eab4cf3-570b-4827-b25d-c01a5683a048","Type":"ContainerDied","Data":"ca04bc95c46b7fbca715653bcb98dea98ff7e87e45fd86c7b9ed9c373330c6a0"}
Nov 02 21:14:42.692319 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.692277    5224 scope.go:122] "RemoveContainer" containerID="7d80df9b2ec74d6f25a367c6b48dd84a284c5acc7bbc90286fb9c7ef32d3b0c5"
Nov 02 21:14:42.701722 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.701706    5224 scope.go:122] "RemoveContainer" containerID="7d80df9b2ec74d6f25a367c6b48dd84a284c5acc7bbc90286fb9c7ef32d3b0c5"
Nov 02 21:14:42.702017 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:14:42.701998    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d80df9b2ec74d6f25a367c6b48dd84a284c5acc7bbc90286fb9c7ef32d3b0c5\": container with ID starting with 7d80df9b2ec74d6f25a367c6b48dd84a284c5acc7bbc90286fb9c7ef32d3b0c5 not found: ID does not exist" containerID="7d80df9b2ec74d6f25a367c6b48dd84a284c5acc7bbc90286fb9c7ef32d3b0c5"
Nov 02 21:14:42.702073 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.702027    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d80df9b2ec74d6f25a367c6b48dd84a284c5acc7bbc90286fb9c7ef32d3b0c5"} err="failed to get container status \"7d80df9b2ec74d6f25a367c6b48dd84a284c5acc7bbc90286fb9c7ef32d3b0c5\": rpc error: code = NotFound desc = could not find container \"7d80df9b2ec74d6f25a367c6b48dd84a284c5acc7bbc90286fb9c7ef32d3b0c5\": container with ID starting with 7d80df9b2ec74d6f25a367c6b48dd84a284c5acc7bbc90286fb9c7ef32d3b0c5 not found: ID does not exist"
Nov 02 21:14:42.860520 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.860482    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"]
Nov 02 21:14:42.865913 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:42.865884    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-9rmzr"]
Nov 02 21:14:43.166131 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:43.166097    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eab4cf3-570b-4827-b25d-c01a5683a048" path="/var/lib/kubelet/pods/0eab4cf3-570b-4827-b25d-c01a5683a048/volumes"
Nov 02 21:14:44.155488 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:44.155455    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:44.533099 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:44.533061    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:48.990953 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:48.990920    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:50.431021 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:50.430972    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:51.330610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:51.330576    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"]
Nov 02 21:14:51.616628 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:51.616536    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:51.791912 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:51.791871    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:54.208541 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:54.207782    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:55.729723 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:55.729689    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:14:56.331781 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:56.331734    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t" podUID="509e4240-8ed0-4980-90bc-53c5c21ffea8" containerName="payload-pre-processing" containerID="cri-o://767c1f960ad5deb32bcff979ca10eaab72e2ff5d598f879b7edab76f00fe0310" gracePeriod=25
Nov 02 21:14:56.377287 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:56.377257    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:14:57.357752 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:14:57.357722    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:14:57.505191 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.505148    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6" podUID="675c9473-71e0-4bd1-8804-e684edf052fb" containerName="main" containerID="cri-o://a386d8ce57d4281fd0aef3867beb5f7342048fc6c771da4a27cbc43b15e59125" gracePeriod=45
Nov 02 21:14:57.751020 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.750991    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 21:14:57.782848 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.782813    5224 generic.go:356] "Generic (PLEG): container finished" podID="675c9473-71e0-4bd1-8804-e684edf052fb" containerID="a386d8ce57d4281fd0aef3867beb5f7342048fc6c771da4a27cbc43b15e59125" exitCode=0
Nov 02 21:14:57.782992 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.782861    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"
Nov 02 21:14:57.782992 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.782857    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6" event={"ID":"675c9473-71e0-4bd1-8804-e684edf052fb","Type":"ContainerDied","Data":"a386d8ce57d4281fd0aef3867beb5f7342048fc6c771da4a27cbc43b15e59125"}
Nov 02 21:14:57.782992 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.782984    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6" event={"ID":"675c9473-71e0-4bd1-8804-e684edf052fb","Type":"ContainerDied","Data":"aa70efff6531fe8b30859c1cdfccd8997baf625b62e657e1583f4ba95a051c91"}
Nov 02 21:14:57.783102 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.783009    5224 scope.go:122] "RemoveContainer" containerID="a386d8ce57d4281fd0aef3867beb5f7342048fc6c771da4a27cbc43b15e59125"
Nov 02 21:14:57.790985 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.790965    5224 scope.go:122] "RemoveContainer" containerID="a386d8ce57d4281fd0aef3867beb5f7342048fc6c771da4a27cbc43b15e59125"
Nov 02 21:14:57.791272 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:14:57.791251    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a386d8ce57d4281fd0aef3867beb5f7342048fc6c771da4a27cbc43b15e59125\": container with ID starting with a386d8ce57d4281fd0aef3867beb5f7342048fc6c771da4a27cbc43b15e59125 not found: ID does not exist" containerID="a386d8ce57d4281fd0aef3867beb5f7342048fc6c771da4a27cbc43b15e59125"
Nov 02 21:14:57.791328 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.791282    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a386d8ce57d4281fd0aef3867beb5f7342048fc6c771da4a27cbc43b15e59125"} err="failed to get container status \"a386d8ce57d4281fd0aef3867beb5f7342048fc6c771da4a27cbc43b15e59125\": rpc error: code = NotFound desc = could not find container \"a386d8ce57d4281fd0aef3867beb5f7342048fc6c771da4a27cbc43b15e59125\": container with ID starting with a386d8ce57d4281fd0aef3867beb5f7342048fc6c771da4a27cbc43b15e59125 not found: ID does not exist"
Nov 02 21:14:57.880401 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.880333    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675c9473-71e0-4bd1-8804-e684edf052fb-kube-api-access-4grtv\" (UniqueName: \"kubernetes.io/projected/675c9473-71e0-4bd1-8804-e684edf052fb-kube-api-access-4grtv\") pod \"675c9473-71e0-4bd1-8804-e684edf052fb\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") "
Nov 02 21:14:57.880401 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.880376    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-model-cache\") pod \"675c9473-71e0-4bd1-8804-e684edf052fb\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") "
Nov 02 21:14:57.880723 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.880404    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-dshm\") pod \"675c9473-71e0-4bd1-8804-e684edf052fb\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") "
Nov 02 21:14:57.880723 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.880468    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-tmp-dir\") pod \"675c9473-71e0-4bd1-8804-e684edf052fb\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") "
Nov 02 21:14:57.880723 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.880500    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-home\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-home\") pod \"675c9473-71e0-4bd1-8804-e684edf052fb\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") "
Nov 02 21:14:57.880723 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.880538    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/675c9473-71e0-4bd1-8804-e684edf052fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/675c9473-71e0-4bd1-8804-e684edf052fb-tls-certs\") pod \"675c9473-71e0-4bd1-8804-e684edf052fb\" (UID: \"675c9473-71e0-4bd1-8804-e684edf052fb\") "
Nov 02 21:14:57.880723 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.880674    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-model-cache" pod "675c9473-71e0-4bd1-8804-e684edf052fb" (UID: "675c9473-71e0-4bd1-8804-e684edf052fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:14:57.881052 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.880937    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:57.881273 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.881242    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-home" pod "675c9473-71e0-4bd1-8804-e684edf052fb" (UID: "675c9473-71e0-4bd1-8804-e684edf052fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:14:57.881401 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.881320    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-tmp-dir" pod "675c9473-71e0-4bd1-8804-e684edf052fb" (UID: "675c9473-71e0-4bd1-8804-e684edf052fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:14:57.883099 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.883077    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675c9473-71e0-4bd1-8804-e684edf052fb-tls-certs" pod "675c9473-71e0-4bd1-8804-e684edf052fb" (UID: "675c9473-71e0-4bd1-8804-e684edf052fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:14:57.883305 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.883283    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675c9473-71e0-4bd1-8804-e684edf052fb-kube-api-access-4grtv" pod "675c9473-71e0-4bd1-8804-e684edf052fb" (UID: "675c9473-71e0-4bd1-8804-e684edf052fb"). InnerVolumeSpecName "kube-api-access-4grtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:14:57.883416 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.883307    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-dshm" pod "675c9473-71e0-4bd1-8804-e684edf052fb" (UID: "675c9473-71e0-4bd1-8804-e684edf052fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:14:57.982197 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.982169    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4grtv\" (UniqueName: \"kubernetes.io/projected/675c9473-71e0-4bd1-8804-e684edf052fb-kube-api-access-4grtv\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:57.982197 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.982197    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:57.982407 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.982209    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:57.982407 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.982222    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/675c9473-71e0-4bd1-8804-e684edf052fb-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:57.982407 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:57.982235    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/675c9473-71e0-4bd1-8804-e684edf052fb-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:58.205086 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:58.204997    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"]
Nov 02 21:14:58.206835 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:58.206816    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/simulator-w1-bbee67-kserve-d9c686c9-gjgq6"]
Nov 02 21:14:58.811393 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:58.811350    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t" podUID="509e4240-8ed0-4980-90bc-53c5c21ffea8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 21:14:59.166563 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:59.166472    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="675c9473-71e0-4bd1-8804-e684edf052fb" path="/var/lib/kubelet/pods/675c9473-71e0-4bd1-8804-e684edf052fb/volumes"
Nov 02 21:14:59.270732 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:14:59.270696    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:00.163898 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.163862    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"]
Nov 02 21:15:00.164411 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164388    5224 cpu_manager.go:414] "containerMap: removing container" podUID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" containerName="istio-proxy"
Nov 02 21:15:00.164491 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164414    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" containerName="istio-proxy"
Nov 02 21:15:00.164491 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164437    5224 cpu_manager.go:414] "containerMap: removing container" podUID="bc611f03-fc41-41c6-a631-d0b344a7c8f9" containerName="main"
Nov 02 21:15:00.164491 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164445    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc611f03-fc41-41c6-a631-d0b344a7c8f9" containerName="main"
Nov 02 21:15:00.164491 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164466    5224 cpu_manager.go:414] "containerMap: removing container" podUID="675c9473-71e0-4bd1-8804-e684edf052fb" containerName="main"
Nov 02 21:15:00.164491 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164473    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675c9473-71e0-4bd1-8804-e684edf052fb" containerName="main"
Nov 02 21:15:00.164491 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164483    5224 cpu_manager.go:414] "containerMap: removing container" podUID="8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43" containerName="test-curl-7522-9a5907"
Nov 02 21:15:00.164491 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164490    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43" containerName="test-curl-7522-9a5907"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164505    5224 cpu_manager.go:414] "containerMap: removing container" podUID="78e38acc-d4d7-43eb-abc8-323a9723f67e" containerName="payload-processing"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164513    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e38acc-d4d7-43eb-abc8-323a9723f67e" containerName="payload-processing"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164536    5224 cpu_manager.go:414] "containerMap: removing container" podUID="f93a40bb-5ce3-4355-b2d4-6228d7857424" containerName="maas-api"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164544    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93a40bb-5ce3-4355-b2d4-6228d7857424" containerName="maas-api"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164556    5224 cpu_manager.go:414] "containerMap: removing container" podUID="87bb590b-e94c-4051-9df7-09be7673c1e8" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164563    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87bb590b-e94c-4051-9df7-09be7673c1e8" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164573    5224 cpu_manager.go:414] "containerMap: removing container" podUID="3359dc52-961f-43fd-b567-3bcb22271302" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164580    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3359dc52-961f-43fd-b567-3bcb22271302" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164589    5224 cpu_manager.go:414] "containerMap: removing container" podUID="bdb0be65-a5fd-4ea9-8573-d9a755151a44" containerName="maas-api"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164596    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdb0be65-a5fd-4ea9-8573-d9a755151a44" containerName="maas-api"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164605    5224 cpu_manager.go:414] "containerMap: removing container" podUID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" containerName="istio-proxy"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164611    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" containerName="istio-proxy"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164626    5224 cpu_manager.go:414] "containerMap: removing container" podUID="f67432d1-22af-458b-bd57-002a02c62a5d" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164633    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67432d1-22af-458b-bd57-002a02c62a5d" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164647    5224 cpu_manager.go:414] "containerMap: removing container" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164656    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164669    5224 cpu_manager.go:414] "containerMap: removing container" podUID="0eab4cf3-570b-4827-b25d-c01a5683a048" containerName="istio-proxy"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164677    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eab4cf3-570b-4827-b25d-c01a5683a048" containerName="istio-proxy"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164693    5224 cpu_manager.go:414] "containerMap: removing container" podUID="f4ff50fb-2482-4330-9ce4-e05d794c2a2a" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164700    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ff50fb-2482-4330-9ce4-e05d794c2a2a" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164709    5224 cpu_manager.go:414] "containerMap: removing container" podUID="c2b4bca4-928d-439e-b582-979ff1768fc6" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164716    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b4bca4-928d-439e-b582-979ff1768fc6" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164728    5224 cpu_manager.go:414] "containerMap: removing container" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164735    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164750    5224 cpu_manager.go:414] "containerMap: removing container" podUID="7689b623-5d86-473a-82aa-0739d899ff06" containerName="maas-api"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164758    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7689b623-5d86-473a-82aa-0739d899ff06" containerName="maas-api"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164794    5224 cpu_manager.go:414] "containerMap: removing container" podUID="bc2af629-50fa-4966-87d5-d4d399d844b1" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164802    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2af629-50fa-4966-87d5-d4d399d844b1" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164813    5224 cpu_manager.go:414] "containerMap: removing container" podUID="de83a78c-767c-4f9f-a0b4-3560413bbba7" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164820    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de83a78c-767c-4f9f-a0b4-3560413bbba7" containerName="main"
Nov 02 21:15:00.164909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164918    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93a40bb-5ce3-4355-b2d4-6228d7857424" containerName="maas-api"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164934    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="87bb590b-e94c-4051-9df7-09be7673c1e8" containerName="main"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164945    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3359dc52-961f-43fd-b567-3bcb22271302" containerName="main"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164956    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdb0be65-a5fd-4ea9-8573-d9a755151a44" containerName="maas-api"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164966    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f254db-a4e4-4fed-ac7d-d8b61600091e" containerName="istio-proxy"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164976    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67432d1-22af-458b-bd57-002a02c62a5d" containerName="main"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164985    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3542c449-0d85-4534-9284-67c7250ccd64" containerName="istio-proxy"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.164995    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eab4cf3-570b-4827-b25d-c01a5683a048" containerName="istio-proxy"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.165003    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ff50fb-2482-4330-9ce4-e05d794c2a2a" containerName="main"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.165011    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b4bca4-928d-439e-b582-979ff1768fc6" containerName="main"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.165018    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa503409-81d4-4532-9dc8-64a7aa8d0443" containerName="main"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.165028    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7689b623-5d86-473a-82aa-0739d899ff06" containerName="maas-api"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.165037    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2af629-50fa-4966-87d5-d4d399d844b1" containerName="main"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.165046    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="de83a78c-767c-4f9f-a0b4-3560413bbba7" containerName="main"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.165055    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="695b48a2-b9f5-4cb9-8e40-26835cdb5d71" containerName="istio-proxy"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.165065    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc611f03-fc41-41c6-a631-d0b344a7c8f9" containerName="main"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.165075    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="675c9473-71e0-4bd1-8804-e684edf052fb" containerName="main"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.165084    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a691c16-0a1b-43d7-a2d6-6b0d5aa22f43" containerName="test-curl-7522-9a5907"
Nov 02 21:15:00.166396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.165095    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e38acc-d4d7-43eb-abc8-323a9723f67e" containerName="payload-processing"
Nov 02 21:15:00.169164 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.169143    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"]
Nov 02 21:15:00.169275 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.169254    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"
Nov 02 21:15:00.172517 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.172495    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"
Nov 02 21:15:00.201632 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.201373    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnfzp\" (UniqueName: \"kubernetes.io/projected/a18abf8b-0c6a-4d86-9a76-55aaa395298e-kube-api-access-cnfzp\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9\" (UID: \"a18abf8b-0c6a-4d86-9a76-55aaa395298e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"
Nov 02 21:15:00.201632 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.201423    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18abf8b-0c6a-4d86-9a76-55aaa395298e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9\" (UID: \"a18abf8b-0c6a-4d86-9a76-55aaa395298e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"
Nov 02 21:15:00.223357 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.223331    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:15:00.223527 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.223510    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:15:00.275099 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.275072    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 21:15:00.276711 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.276685    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"]
Nov 02 21:15:00.277975 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.277956    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"]
Nov 02 21:15:00.302633 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.302600    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18abf8b-0c6a-4d86-9a76-55aaa395298e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9\" (UID: \"a18abf8b-0c6a-4d86-9a76-55aaa395298e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"
Nov 02 21:15:00.302949 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.302909    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59f8676-3618-4d0b-a026-6461cb0c195f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894235-x4hck\" (UID: \"c59f8676-3618-4d0b-a026-6461cb0c195f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"
Nov 02 21:15:00.303289 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.303259    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlsx6\" (UniqueName: \"kubernetes.io/projected/c59f8676-3618-4d0b-a026-6461cb0c195f-kube-api-access-mlsx6\") pod \"maas-api-key-cleanup-29894235-x4hck\" (UID: \"c59f8676-3618-4d0b-a026-6461cb0c195f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"
Nov 02 21:15:00.303589 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.303568    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18abf8b-0c6a-4d86-9a76-55aaa395298e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9\" (UID: \"a18abf8b-0c6a-4d86-9a76-55aaa395298e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"
Nov 02 21:15:00.303721 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.303640    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnfzp\" (UniqueName: \"kubernetes.io/projected/a18abf8b-0c6a-4d86-9a76-55aaa395298e-kube-api-access-cnfzp\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9\" (UID: \"a18abf8b-0c6a-4d86-9a76-55aaa395298e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"
Nov 02 21:15:00.361716 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.361683    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnfzp\" (UniqueName: \"kubernetes.io/projected/a18abf8b-0c6a-4d86-9a76-55aaa395298e-kube-api-access-cnfzp\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9\" (UID: \"a18abf8b-0c6a-4d86-9a76-55aaa395298e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"
Nov 02 21:15:00.404877 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.404851    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlsx6\" (UniqueName: \"kubernetes.io/projected/c59f8676-3618-4d0b-a026-6461cb0c195f-kube-api-access-mlsx6\") pod \"maas-api-key-cleanup-29894235-x4hck\" (UID: \"c59f8676-3618-4d0b-a026-6461cb0c195f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"
Nov 02 21:15:00.405004 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.404947    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59f8676-3618-4d0b-a026-6461cb0c195f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894235-x4hck\" (UID: \"c59f8676-3618-4d0b-a026-6461cb0c195f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"
Nov 02 21:15:00.405481 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.405462    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59f8676-3618-4d0b-a026-6461cb0c195f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894235-x4hck\" (UID: \"c59f8676-3618-4d0b-a026-6461cb0c195f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"
Nov 02 21:15:00.465089 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.465014    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlsx6\" (UniqueName: \"kubernetes.io/projected/c59f8676-3618-4d0b-a026-6461cb0c195f-kube-api-access-mlsx6\") pod \"maas-api-key-cleanup-29894235-x4hck\" (UID: \"c59f8676-3618-4d0b-a026-6461cb0c195f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"
Nov 02 21:15:00.486543 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.482833    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"
Nov 02 21:15:00.489414 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.489397    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"
Nov 02 21:15:00.848631 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:15:00.848595    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59f8676_3618_4d0b_a026_6461cb0c195f.slice/crio-ef1bed0afc05135f9d9dd70339a80421590f358463e2582ecc03c0d57af4a132 WatchSource:0}: Error finding container ef1bed0afc05135f9d9dd70339a80421590f358463e2582ecc03c0d57af4a132: Status 404 returned error can't find the container with id ef1bed0afc05135f9d9dd70339a80421590f358463e2582ecc03c0d57af4a132
Nov 02 21:15:00.852434 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.852279    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"]
Nov 02 21:15:00.893751 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:15:00.893725    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18abf8b_0c6a_4d86_9a76_55aaa395298e.slice/crio-670866e37d546b226bb09bb5601efcd7bda69525bca88553e2e431cfb629e7e3 WatchSource:0}: Error finding container 670866e37d546b226bb09bb5601efcd7bda69525bca88553e2e431cfb629e7e3: Status 404 returned error can't find the container with id 670866e37d546b226bb09bb5601efcd7bda69525bca88553e2e431cfb629e7e3
Nov 02 21:15:00.897624 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:00.897539    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"]
Nov 02 21:15:01.805987 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:01.805953    5224 generic.go:356] "Generic (PLEG): container finished" podID="a18abf8b-0c6a-4d86-9a76-55aaa395298e" containerID="f00e8923d543fb03708c9d9e63552504f9da0099166a3de49f00e178d0616bc9" exitCode=60
Nov 02 21:15:01.806435 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:01.806036    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9" event={"ID":"a18abf8b-0c6a-4d86-9a76-55aaa395298e","Type":"ContainerDied","Data":"f00e8923d543fb03708c9d9e63552504f9da0099166a3de49f00e178d0616bc9"}
Nov 02 21:15:01.806435 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:01.806092    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9" event={"ID":"a18abf8b-0c6a-4d86-9a76-55aaa395298e","Type":"ContainerStarted","Data":"670866e37d546b226bb09bb5601efcd7bda69525bca88553e2e431cfb629e7e3"}
Nov 02 21:15:01.806435 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:01.806256    5224 scope.go:122] "RemoveContainer" containerID="f00e8923d543fb03708c9d9e63552504f9da0099166a3de49f00e178d0616bc9"
Nov 02 21:15:01.807743 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:01.807686    5224 generic.go:356] "Generic (PLEG): container finished" podID="c59f8676-3618-4d0b-a026-6461cb0c195f" containerID="f199cfb8d5822b9abf841098422712f93cecbae31cae0d29da47d15d44e54e50" exitCode=60
Nov 02 21:15:01.807850 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:01.807737    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck" event={"ID":"c59f8676-3618-4d0b-a026-6461cb0c195f","Type":"ContainerDied","Data":"f199cfb8d5822b9abf841098422712f93cecbae31cae0d29da47d15d44e54e50"}
Nov 02 21:15:01.807850 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:01.807788    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck" event={"ID":"c59f8676-3618-4d0b-a026-6461cb0c195f","Type":"ContainerStarted","Data":"ef1bed0afc05135f9d9dd70339a80421590f358463e2582ecc03c0d57af4a132"}
Nov 02 21:15:01.808011 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:01.807994    5224 scope.go:122] "RemoveContainer" containerID="f199cfb8d5822b9abf841098422712f93cecbae31cae0d29da47d15d44e54e50"
Nov 02 21:15:02.171155 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:02.171078    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:02.814685 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:02.814599    5224 generic.go:356] "Generic (PLEG): container finished" podID="c59f8676-3618-4d0b-a026-6461cb0c195f" containerID="bd92f120dd4ba5239f7f8a112907933fa85c202265a92c9c06dc484c113441f9" exitCode=60
Nov 02 21:15:02.815235 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:02.814683    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck" event={"ID":"c59f8676-3618-4d0b-a026-6461cb0c195f","Type":"ContainerDied","Data":"bd92f120dd4ba5239f7f8a112907933fa85c202265a92c9c06dc484c113441f9"}
Nov 02 21:15:02.815235 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:02.814740    5224 scope.go:122] "RemoveContainer" containerID="f199cfb8d5822b9abf841098422712f93cecbae31cae0d29da47d15d44e54e50"
Nov 02 21:15:02.815235 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:02.815009    5224 scope.go:122] "RemoveContainer" containerID="bd92f120dd4ba5239f7f8a112907933fa85c202265a92c9c06dc484c113441f9"
Nov 02 21:15:02.815416 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:15:02.815257    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894235-x4hck_odh-ai-gateway-infra(c59f8676-3618-4d0b-a026-6461cb0c195f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck" podUID="c59f8676-3618-4d0b-a026-6461cb0c195f"
Nov 02 21:15:02.816719 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:02.816696    5224 generic.go:356] "Generic (PLEG): container finished" podID="a18abf8b-0c6a-4d86-9a76-55aaa395298e" containerID="8eea79fd108dd86265be3fcd95e52caf73c74f8f82c60895003db3e5e7775479" exitCode=60
Nov 02 21:15:02.816841 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:02.816780    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9" event={"ID":"a18abf8b-0c6a-4d86-9a76-55aaa395298e","Type":"ContainerDied","Data":"8eea79fd108dd86265be3fcd95e52caf73c74f8f82c60895003db3e5e7775479"}
Nov 02 21:15:02.817052 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:02.817039    5224 scope.go:122] "RemoveContainer" containerID="8eea79fd108dd86265be3fcd95e52caf73c74f8f82c60895003db3e5e7775479"
Nov 02 21:15:02.817229 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:15:02.817215    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9_odh-ai-gateway-infra(a18abf8b-0c6a-4d86-9a76-55aaa395298e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9" podUID="a18abf8b-0c6a-4d86-9a76-55aaa395298e"
Nov 02 21:15:02.826141 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:02.826123    5224 scope.go:122] "RemoveContainer" containerID="f00e8923d543fb03708c9d9e63552504f9da0099166a3de49f00e178d0616bc9"
Nov 02 21:15:03.824653 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:03.824622    5224 scope.go:122] "RemoveContainer" containerID="bd92f120dd4ba5239f7f8a112907933fa85c202265a92c9c06dc484c113441f9"
Nov 02 21:15:03.825129 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:15:03.824904    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894235-x4hck_odh-ai-gateway-infra(c59f8676-3618-4d0b-a026-6461cb0c195f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck" podUID="c59f8676-3618-4d0b-a026-6461cb0c195f"
Nov 02 21:15:03.826179 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:03.826161    5224 scope.go:122] "RemoveContainer" containerID="8eea79fd108dd86265be3fcd95e52caf73c74f8f82c60895003db3e5e7775479"
Nov 02 21:15:03.826370 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:15:03.826352    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9_odh-ai-gateway-infra(a18abf8b-0c6a-4d86-9a76-55aaa395298e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9" podUID="a18abf8b-0c6a-4d86-9a76-55aaa395298e"
Nov 02 21:15:04.162678 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:04.162580    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:08.811326 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:08.811288    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t" podUID="509e4240-8ed0-4980-90bc-53c5c21ffea8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 21:15:09.034719 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:09.034682    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:13.825082 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:13.825046    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:14.054127 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:14.054090    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:14.160664 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:14.160577    5224 scope.go:122] "RemoveContainer" containerID="bd92f120dd4ba5239f7f8a112907933fa85c202265a92c9c06dc484c113441f9"
Nov 02 21:15:14.540040 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:14.540004    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:14.879567 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:14.879529    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck" event={"ID":"c59f8676-3618-4d0b-a026-6461cb0c195f","Type":"ContainerStarted","Data":"07018e897e4b684f3ebd5511f77f778360a35f35ead1cd0bbd5945ccb5705248"}
Nov 02 21:15:14.996124 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:14.996072    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck" podStartSLOduration=14.996057826 podStartE2EDuration="14.996057826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:14.994271462 +0000 UTC m=+5195.991960071" watchObservedRunningTime="2026-11-02 21:15:14.996057826 +0000 UTC m=+5195.993746432"
Nov 02 21:15:15.161254 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:15.161171    5224 scope.go:122] "RemoveContainer" containerID="8eea79fd108dd86265be3fcd95e52caf73c74f8f82c60895003db3e5e7775479"
Nov 02 21:15:15.886368 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:15.886336    5224 generic.go:356] "Generic (PLEG): container finished" podID="a18abf8b-0c6a-4d86-9a76-55aaa395298e" containerID="5e3b458cc42a50a0cf51f894b728eb04181786b41e67d26df6e87d6e4cc69d87" exitCode=60
Nov 02 21:15:15.886838 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:15.886408    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9" event={"ID":"a18abf8b-0c6a-4d86-9a76-55aaa395298e","Type":"ContainerDied","Data":"5e3b458cc42a50a0cf51f894b728eb04181786b41e67d26df6e87d6e4cc69d87"}
Nov 02 21:15:15.886838 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:15.886472    5224 scope.go:122] "RemoveContainer" containerID="8eea79fd108dd86265be3fcd95e52caf73c74f8f82c60895003db3e5e7775479"
Nov 02 21:15:15.886962 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:15.886850    5224 scope.go:122] "RemoveContainer" containerID="5e3b458cc42a50a0cf51f894b728eb04181786b41e67d26df6e87d6e4cc69d87"
Nov 02 21:15:15.887125 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:15:15.887106    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9_odh-ai-gateway-infra(a18abf8b-0c6a-4d86-9a76-55aaa395298e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9" podUID="a18abf8b-0c6a-4d86-9a76-55aaa395298e"
Nov 02 21:15:15.888504 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:15.888485    5224 generic.go:356] "Generic (PLEG): container finished" podID="c59f8676-3618-4d0b-a026-6461cb0c195f" containerID="07018e897e4b684f3ebd5511f77f778360a35f35ead1cd0bbd5945ccb5705248" exitCode=60
Nov 02 21:15:15.888586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:15.888532    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck" event={"ID":"c59f8676-3618-4d0b-a026-6461cb0c195f","Type":"ContainerDied","Data":"07018e897e4b684f3ebd5511f77f778360a35f35ead1cd0bbd5945ccb5705248"}
Nov 02 21:15:15.888846 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:15.888832    5224 scope.go:122] "RemoveContainer" containerID="07018e897e4b684f3ebd5511f77f778360a35f35ead1cd0bbd5945ccb5705248"
Nov 02 21:15:15.889044 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:15:15.889029    5224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894235-x4hck_odh-ai-gateway-infra(c59f8676-3618-4d0b-a026-6461cb0c195f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck" podUID="c59f8676-3618-4d0b-a026-6461cb0c195f"
Nov 02 21:15:15.931874 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:15.931843    5224 scope.go:122] "RemoveContainer" containerID="bd92f120dd4ba5239f7f8a112907933fa85c202265a92c9c06dc484c113441f9"
Nov 02 21:15:16.008203 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.008170    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"]
Nov 02 21:15:16.461270 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.461229    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:16.580523 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.580495    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"
Nov 02 21:15:16.654203 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.654109    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/509e4240-8ed0-4980-90bc-53c5c21ffea8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/509e4240-8ed0-4980-90bc-53c5c21ffea8-plugins-config-volume\") pod \"509e4240-8ed0-4980-90bc-53c5c21ffea8\" (UID: \"509e4240-8ed0-4980-90bc-53c5c21ffea8\") "
Nov 02 21:15:16.654203 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.654166    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/509e4240-8ed0-4980-90bc-53c5c21ffea8-kube-api-access-vvr8z\" (UniqueName: \"kubernetes.io/projected/509e4240-8ed0-4980-90bc-53c5c21ffea8-kube-api-access-vvr8z\") pod \"509e4240-8ed0-4980-90bc-53c5c21ffea8\" (UID: \"509e4240-8ed0-4980-90bc-53c5c21ffea8\") "
Nov 02 21:15:16.654524 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.654501    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/509e4240-8ed0-4980-90bc-53c5c21ffea8-plugins-config-volume" pod "509e4240-8ed0-4980-90bc-53c5c21ffea8" (UID: "509e4240-8ed0-4980-90bc-53c5c21ffea8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:15:16.656389 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.656364    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509e4240-8ed0-4980-90bc-53c5c21ffea8-kube-api-access-vvr8z" pod "509e4240-8ed0-4980-90bc-53c5c21ffea8" (UID: "509e4240-8ed0-4980-90bc-53c5c21ffea8"). InnerVolumeSpecName "kube-api-access-vvr8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:15:16.755512 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.755474    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvr8z\" (UniqueName: \"kubernetes.io/projected/509e4240-8ed0-4980-90bc-53c5c21ffea8-kube-api-access-vvr8z\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:16.755512 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.755508    5224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/509e4240-8ed0-4980-90bc-53c5c21ffea8-plugins-config-volume\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:16.894433 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.894395    5224 generic.go:356] "Generic (PLEG): container finished" podID="509e4240-8ed0-4980-90bc-53c5c21ffea8" containerID="767c1f960ad5deb32bcff979ca10eaab72e2ff5d598f879b7edab76f00fe0310" exitCode=0
Nov 02 21:15:16.894981 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.894439    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"
Nov 02 21:15:16.894981 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.894485    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t" event={"ID":"509e4240-8ed0-4980-90bc-53c5c21ffea8","Type":"ContainerDied","Data":"767c1f960ad5deb32bcff979ca10eaab72e2ff5d598f879b7edab76f00fe0310"}
Nov 02 21:15:16.894981 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.894538    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t" event={"ID":"509e4240-8ed0-4980-90bc-53c5c21ffea8","Type":"ContainerDied","Data":"e64fbc84e010b82653a3b91cbb085b9d8637902f905384c99b9c0eb6bf422c6e"}
Nov 02 21:15:16.894981 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.894566    5224 scope.go:122] "RemoveContainer" containerID="767c1f960ad5deb32bcff979ca10eaab72e2ff5d598f879b7edab76f00fe0310"
Nov 02 21:15:16.904154 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.904128    5224 scope.go:122] "RemoveContainer" containerID="767c1f960ad5deb32bcff979ca10eaab72e2ff5d598f879b7edab76f00fe0310"
Nov 02 21:15:16.904489 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:15:16.904442    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767c1f960ad5deb32bcff979ca10eaab72e2ff5d598f879b7edab76f00fe0310\": container with ID starting with 767c1f960ad5deb32bcff979ca10eaab72e2ff5d598f879b7edab76f00fe0310 not found: ID does not exist" containerID="767c1f960ad5deb32bcff979ca10eaab72e2ff5d598f879b7edab76f00fe0310"
Nov 02 21:15:16.904489 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:16.904476    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767c1f960ad5deb32bcff979ca10eaab72e2ff5d598f879b7edab76f00fe0310"} err="failed to get container status \"767c1f960ad5deb32bcff979ca10eaab72e2ff5d598f879b7edab76f00fe0310\": rpc error: code = NotFound desc = could not find container \"767c1f960ad5deb32bcff979ca10eaab72e2ff5d598f879b7edab76f00fe0310\": container with ID starting with 767c1f960ad5deb32bcff979ca10eaab72e2ff5d598f879b7edab76f00fe0310 not found: ID does not exist"
Nov 02 21:15:17.010878 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.010846    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"]
Nov 02 21:15:17.040333 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.039427    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"
Nov 02 21:15:17.060578 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.060541    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"]
Nov 02 21:15:17.062499 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.062479    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-bbee67-797b77bdb7-sj79t"]
Nov 02 21:15:17.142232 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.142203    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"
Nov 02 21:15:17.159350 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.159265    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59f8676-3618-4d0b-a026-6461cb0c195f-kube-api-access-mlsx6\" (UniqueName: \"kubernetes.io/projected/c59f8676-3618-4d0b-a026-6461cb0c195f-kube-api-access-mlsx6\") pod \"c59f8676-3618-4d0b-a026-6461cb0c195f\" (UID: \"c59f8676-3618-4d0b-a026-6461cb0c195f\") "
Nov 02 21:15:17.159350 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.159339    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c59f8676-3618-4d0b-a026-6461cb0c195f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59f8676-3618-4d0b-a026-6461cb0c195f-maas-api-service-ca\") pod \"c59f8676-3618-4d0b-a026-6461cb0c195f\" (UID: \"c59f8676-3618-4d0b-a026-6461cb0c195f\") "
Nov 02 21:15:17.159688 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.159665    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59f8676-3618-4d0b-a026-6461cb0c195f-maas-api-service-ca" pod "c59f8676-3618-4d0b-a026-6461cb0c195f" (UID: "c59f8676-3618-4d0b-a026-6461cb0c195f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:15:17.161421 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.161400    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59f8676-3618-4d0b-a026-6461cb0c195f-kube-api-access-mlsx6" pod "c59f8676-3618-4d0b-a026-6461cb0c195f" (UID: "c59f8676-3618-4d0b-a026-6461cb0c195f"). InnerVolumeSpecName "kube-api-access-mlsx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:15:17.165974 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.165950    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="509e4240-8ed0-4980-90bc-53c5c21ffea8" path="/var/lib/kubelet/pods/509e4240-8ed0-4980-90bc-53c5c21ffea8/volumes"
Nov 02 21:15:17.260789 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.260744    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18abf8b-0c6a-4d86-9a76-55aaa395298e-kube-api-access-cnfzp\" (UniqueName: \"kubernetes.io/projected/a18abf8b-0c6a-4d86-9a76-55aaa395298e-kube-api-access-cnfzp\") pod \"a18abf8b-0c6a-4d86-9a76-55aaa395298e\" (UID: \"a18abf8b-0c6a-4d86-9a76-55aaa395298e\") "
Nov 02 21:15:17.261000 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.260841    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18abf8b-0c6a-4d86-9a76-55aaa395298e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18abf8b-0c6a-4d86-9a76-55aaa395298e-maas-api-service-ca\") pod \"a18abf8b-0c6a-4d86-9a76-55aaa395298e\" (UID: \"a18abf8b-0c6a-4d86-9a76-55aaa395298e\") "
Nov 02 21:15:17.261122 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.261103    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlsx6\" (UniqueName: \"kubernetes.io/projected/c59f8676-3618-4d0b-a026-6461cb0c195f-kube-api-access-mlsx6\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:17.261202 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.261144    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59f8676-3618-4d0b-a026-6461cb0c195f-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:17.261261 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.261232    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18abf8b-0c6a-4d86-9a76-55aaa395298e-maas-api-service-ca" pod "a18abf8b-0c6a-4d86-9a76-55aaa395298e" (UID: "a18abf8b-0c6a-4d86-9a76-55aaa395298e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:15:17.263330 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.263302    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18abf8b-0c6a-4d86-9a76-55aaa395298e-kube-api-access-cnfzp" pod "a18abf8b-0c6a-4d86-9a76-55aaa395298e" (UID: "a18abf8b-0c6a-4d86-9a76-55aaa395298e"). InnerVolumeSpecName "kube-api-access-cnfzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:15:17.361760 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.361719    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnfzp\" (UniqueName: \"kubernetes.io/projected/a18abf8b-0c6a-4d86-9a76-55aaa395298e-kube-api-access-cnfzp\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:17.361760 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.361751    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18abf8b-0c6a-4d86-9a76-55aaa395298e-maas-api-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:17.904189 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.904153    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"
Nov 02 21:15:17.904189 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.904176    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck" event={"ID":"c59f8676-3618-4d0b-a026-6461cb0c195f","Type":"ContainerDied","Data":"ef1bed0afc05135f9d9dd70339a80421590f358463e2582ecc03c0d57af4a132"}
Nov 02 21:15:17.904798 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.904234    5224 scope.go:122] "RemoveContainer" containerID="07018e897e4b684f3ebd5511f77f778360a35f35ead1cd0bbd5945ccb5705248"
Nov 02 21:15:17.905538 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.905524    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"
Nov 02 21:15:17.905610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.905566    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9" event={"ID":"a18abf8b-0c6a-4d86-9a76-55aaa395298e","Type":"ContainerDied","Data":"670866e37d546b226bb09bb5601efcd7bda69525bca88553e2e431cfb629e7e3"}
Nov 02 21:15:17.913235 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:17.913215    5224 scope.go:122] "RemoveContainer" containerID="5e3b458cc42a50a0cf51f894b728eb04181786b41e67d26df6e87d6e4cc69d87"
Nov 02 21:15:18.075155 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:18.075123    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"]
Nov 02 21:15:18.077467 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:18.077439    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894235-cbnf9"]
Nov 02 21:15:18.189490 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:18.189407    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"]
Nov 02 21:15:18.194540 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:18.194508    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894235-x4hck"]
Nov 02 21:15:18.706575 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:18.706540    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:18.940428 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:18.940388    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:19.166288 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:19.166252    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18abf8b-0c6a-4d86-9a76-55aaa395298e" path="/var/lib/kubelet/pods/a18abf8b-0c6a-4d86-9a76-55aaa395298e/volumes"
Nov 02 21:15:19.166660 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:19.166646    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c59f8676-3618-4d0b-a026-6461cb0c195f" path="/var/lib/kubelet/pods/c59f8676-3618-4d0b-a026-6461cb0c195f/volumes"
Nov 02 21:15:20.118727 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:20.118690    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:22.995637 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:22.995603    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:23.149390 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:23.149350    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:15:24.812088 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:24.812045    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:25.434680 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:25.434644    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:25.623973 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:25.623936    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:26.142299 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:26.142266    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:26.438839 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:26.438727    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:26.982417 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:26.982385    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:27.365907 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:15:27.365871    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:15:27.442941 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:27.442913    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:30.047057 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.046997    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:30.356467 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.356376    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:30.396080 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396041    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"]
Nov 02 21:15:30.396504 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396486    5224 cpu_manager.go:414] "containerMap: removing container" podUID="c59f8676-3618-4d0b-a026-6461cb0c195f" containerName="cleanup"
Nov 02 21:15:30.396610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396507    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59f8676-3618-4d0b-a026-6461cb0c195f" containerName="cleanup"
Nov 02 21:15:30.396610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396522    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a18abf8b-0c6a-4d86-9a76-55aaa395298e" containerName="cleanup"
Nov 02 21:15:30.396610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396529    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18abf8b-0c6a-4d86-9a76-55aaa395298e" containerName="cleanup"
Nov 02 21:15:30.396610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396544    5224 cpu_manager.go:414] "containerMap: removing container" podUID="c59f8676-3618-4d0b-a026-6461cb0c195f" containerName="cleanup"
Nov 02 21:15:30.396610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396553    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59f8676-3618-4d0b-a026-6461cb0c195f" containerName="cleanup"
Nov 02 21:15:30.396610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396581    5224 cpu_manager.go:414] "containerMap: removing container" podUID="509e4240-8ed0-4980-90bc-53c5c21ffea8" containerName="payload-pre-processing"
Nov 02 21:15:30.396610 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396589    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="509e4240-8ed0-4980-90bc-53c5c21ffea8" containerName="payload-pre-processing"
Nov 02 21:15:30.397032 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396616    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a18abf8b-0c6a-4d86-9a76-55aaa395298e" containerName="cleanup"
Nov 02 21:15:30.397032 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396623    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18abf8b-0c6a-4d86-9a76-55aaa395298e" containerName="cleanup"
Nov 02 21:15:30.397032 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396637    5224 cpu_manager.go:414] "containerMap: removing container" podUID="c59f8676-3618-4d0b-a026-6461cb0c195f" containerName="cleanup"
Nov 02 21:15:30.397032 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396645    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59f8676-3618-4d0b-a026-6461cb0c195f" containerName="cleanup"
Nov 02 21:15:30.397032 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396728    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59f8676-3618-4d0b-a026-6461cb0c195f" containerName="cleanup"
Nov 02 21:15:30.397032 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396740    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18abf8b-0c6a-4d86-9a76-55aaa395298e" containerName="cleanup"
Nov 02 21:15:30.397032 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396751    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59f8676-3618-4d0b-a026-6461cb0c195f" containerName="cleanup"
Nov 02 21:15:30.397032 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396763    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="509e4240-8ed0-4980-90bc-53c5c21ffea8" containerName="payload-pre-processing"
Nov 02 21:15:30.397032 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.396792    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18abf8b-0c6a-4d86-9a76-55aaa395298e" containerName="cleanup"
Nov 02 21:15:30.399830 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.399807    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"
Nov 02 21:15:30.484428 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.484384    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c11b4cae-pxsc4\" (UID: \"08fbda88-2e0b-4119-8fb4-17480f431ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"
Nov 02 21:15:30.484428 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.484429    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c11b4cae-pxsc4\" (UID: \"08fbda88-2e0b-4119-8fb4-17480f431ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"
Nov 02 21:15:30.484722 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.484508    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7mjj\" (UniqueName: \"kubernetes.io/projected/08fbda88-2e0b-4119-8fb4-17480f431ed9-kube-api-access-w7mjj\") pod \"maas-api-revoke-sub-c11b4cae-pxsc4\" (UID: \"08fbda88-2e0b-4119-8fb4-17480f431ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"
Nov 02 21:15:30.505376 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.505334    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:15:30.505376 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.505343    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:15:30.505376 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.505370    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\""
Nov 02 21:15:30.505654 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.505386    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:15:30.506905 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.506885    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"]
Nov 02 21:15:30.529016 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.528983    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:30.585555 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.585517    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c11b4cae-pxsc4\" (UID: \"08fbda88-2e0b-4119-8fb4-17480f431ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"
Nov 02 21:15:30.585555 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.585562    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c11b4cae-pxsc4\" (UID: \"08fbda88-2e0b-4119-8fb4-17480f431ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"
Nov 02 21:15:30.585854 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.585711    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7mjj\" (UniqueName: \"kubernetes.io/projected/08fbda88-2e0b-4119-8fb4-17480f431ed9-kube-api-access-w7mjj\") pod \"maas-api-revoke-sub-c11b4cae-pxsc4\" (UID: \"08fbda88-2e0b-4119-8fb4-17480f431ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"
Nov 02 21:15:30.586342 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.586319    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c11b4cae-pxsc4\" (UID: \"08fbda88-2e0b-4119-8fb4-17480f431ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"
Nov 02 21:15:30.588042 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.588023    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c11b4cae-pxsc4\" (UID: \"08fbda88-2e0b-4119-8fb4-17480f431ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"
Nov 02 21:15:30.643693 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.643616    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7mjj\" (UniqueName: \"kubernetes.io/projected/08fbda88-2e0b-4119-8fb4-17480f431ed9-kube-api-access-w7mjj\") pod \"maas-api-revoke-sub-c11b4cae-pxsc4\" (UID: \"08fbda88-2e0b-4119-8fb4-17480f431ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"
Nov 02 21:15:30.713294 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:30.713246    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"
Nov 02 21:15:31.082917 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:15:31.082883    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fbda88_2e0b_4119_8fb4_17480f431ed9.slice/crio-0e860b94307921871109aa88053820811bc9692b3517c85ccbf68c0b090af2e0 WatchSource:0}: Error finding container 0e860b94307921871109aa88053820811bc9692b3517c85ccbf68c0b090af2e0: Status 404 returned error can't find the container with id 0e860b94307921871109aa88053820811bc9692b3517c85ccbf68c0b090af2e0
Nov 02 21:15:31.086035 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:31.086012    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"]
Nov 02 21:15:31.988198 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:31.988164    5224 generic.go:356] "Generic (PLEG): container finished" podID="08fbda88-2e0b-4119-8fb4-17480f431ed9" containerID="ad2cde5d7a1ad234463cae92068b87fd27445aa9525f3ff8bbabd5975620d7b1" exitCode=0
Nov 02 21:15:31.988377 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:31.988250    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4" event={"ID":"08fbda88-2e0b-4119-8fb4-17480f431ed9","Type":"ContainerDied","Data":"ad2cde5d7a1ad234463cae92068b87fd27445aa9525f3ff8bbabd5975620d7b1"}
Nov 02 21:15:31.988377 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:31.988297    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4" event={"ID":"08fbda88-2e0b-4119-8fb4-17480f431ed9","Type":"ContainerStarted","Data":"0e860b94307921871109aa88053820811bc9692b3517c85ccbf68c0b090af2e0"}
Nov 02 21:15:33.128985 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:33.128955    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"
Nov 02 21:15:33.213233 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:33.213199    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-xks-ca\") pod \"08fbda88-2e0b-4119-8fb4-17480f431ed9\" (UID: \"08fbda88-2e0b-4119-8fb4-17480f431ed9\") "
Nov 02 21:15:33.213233 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:33.213243    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-openshift-ca\") pod \"08fbda88-2e0b-4119-8fb4-17480f431ed9\" (UID: \"08fbda88-2e0b-4119-8fb4-17480f431ed9\") "
Nov 02 21:15:33.213505 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:33.213275    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08fbda88-2e0b-4119-8fb4-17480f431ed9-kube-api-access-w7mjj\" (UniqueName: \"kubernetes.io/projected/08fbda88-2e0b-4119-8fb4-17480f431ed9-kube-api-access-w7mjj\") pod \"08fbda88-2e0b-4119-8fb4-17480f431ed9\" (UID: \"08fbda88-2e0b-4119-8fb4-17480f431ed9\") "
Nov 02 21:15:33.213724 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:33.213695    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-openshift-ca" pod "08fbda88-2e0b-4119-8fb4-17480f431ed9" (UID: "08fbda88-2e0b-4119-8fb4-17480f431ed9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:15:33.215434 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:33.215406    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-xks-ca" pod "08fbda88-2e0b-4119-8fb4-17480f431ed9" (UID: "08fbda88-2e0b-4119-8fb4-17480f431ed9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:15:33.215642 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:33.215470    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fbda88-2e0b-4119-8fb4-17480f431ed9-kube-api-access-w7mjj" pod "08fbda88-2e0b-4119-8fb4-17480f431ed9" (UID: "08fbda88-2e0b-4119-8fb4-17480f431ed9"). InnerVolumeSpecName "kube-api-access-w7mjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:15:33.314576 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:33.314538    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:33.314576 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:33.314570    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fbda88-2e0b-4119-8fb4-17480f431ed9-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:33.314576 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:33.314581    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7mjj\" (UniqueName: \"kubernetes.io/projected/08fbda88-2e0b-4119-8fb4-17480f431ed9-kube-api-access-w7mjj\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:33.999951 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:33.999923    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"
Nov 02 21:15:34.000119 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:33.999917    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4" event={"ID":"08fbda88-2e0b-4119-8fb4-17480f431ed9","Type":"ContainerDied","Data":"0e860b94307921871109aa88053820811bc9692b3517c85ccbf68c0b090af2e0"}
Nov 02 21:15:34.000119 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:34.000081    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e860b94307921871109aa88053820811bc9692b3517c85ccbf68c0b090af2e0"
Nov 02 21:15:35.075028 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:35.074967    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:40.118803 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:40.118754    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:44.986266 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:44.986224    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:45.583091 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:45.583048    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:15:49.782904 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:49.782831    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:54.943237 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:54.943193    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:15:57.358083 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:15:57.358047    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:15:59.877412 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:15:59.877373    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:03.966515 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:03.966483    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:16:04.844683 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:04.844651    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:07.480164 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:07.480124    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"]
Nov 02 21:16:07.480694 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:07.480409    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" podUID="7456949c-c902-4940-b3e4-64fab197654b" containerName="istio-proxy" containerID="cri-o://b16b9f23d4631b84f9d81e750f1f82ee57b8c62d3f87415048b057fdd259cc09" gracePeriod=30
Nov 02 21:16:07.812942 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:07.812901    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d46vl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:16:07.813152 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:07.812982    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" podUID="7456949c-c902-4940-b3e4-64fab197654b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:16:09.681199 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:09.681155    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:12.728100 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.728071    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 21:16:12.883703 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.883670    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-certs\") pod \"7456949c-c902-4940-b3e4-64fab197654b\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") "
Nov 02 21:16:12.883924 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.883723    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-envoy\") pod \"7456949c-c902-4940-b3e4-64fab197654b\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") "
Nov 02 21:16:12.883924 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.883751    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7456949c-c902-4940-b3e4-64fab197654b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7456949c-c902-4940-b3e4-64fab197654b-istiod-ca-cert\") pod \"7456949c-c902-4940-b3e4-64fab197654b\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") "
Nov 02 21:16:12.883924 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.883813    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-kube-api-access-6xlv7\" (UniqueName: \"kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-kube-api-access-6xlv7\") pod \"7456949c-c902-4940-b3e4-64fab197654b\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") "
Nov 02 21:16:12.883924 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.883835    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-data\") pod \"7456949c-c902-4940-b3e4-64fab197654b\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") "
Nov 02 21:16:12.883924 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.883882    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-credential-socket\") pod \"7456949c-c902-4940-b3e4-64fab197654b\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") "
Nov 02 21:16:12.884228 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.884010    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7456949c-c902-4940-b3e4-64fab197654b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7456949c-c902-4940-b3e4-64fab197654b-istio-podinfo\") pod \"7456949c-c902-4940-b3e4-64fab197654b\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") "
Nov 02 21:16:12.884228 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.884027    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-certs" pod "7456949c-c902-4940-b3e4-64fab197654b" (UID: "7456949c-c902-4940-b3e4-64fab197654b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:16:12.884228 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.884065    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-socket\") pod \"7456949c-c902-4940-b3e4-64fab197654b\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") "
Nov 02 21:16:12.884228 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.884118    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-istio-token\" (UniqueName: \"kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-istio-token\") pod \"7456949c-c902-4940-b3e4-64fab197654b\" (UID: \"7456949c-c902-4940-b3e4-64fab197654b\") "
Nov 02 21:16:12.886578 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.884620    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-credential-socket" pod "7456949c-c902-4940-b3e4-64fab197654b" (UID: "7456949c-c902-4940-b3e4-64fab197654b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:16:12.886578 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.885002    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:16:12.886578 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.885028    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:16:12.886578 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.885157    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-data" pod "7456949c-c902-4940-b3e4-64fab197654b" (UID: "7456949c-c902-4940-b3e4-64fab197654b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:16:12.886578 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.885212    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7456949c-c902-4940-b3e4-64fab197654b-istiod-ca-cert" pod "7456949c-c902-4940-b3e4-64fab197654b" (UID: "7456949c-c902-4940-b3e4-64fab197654b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:16:12.886578 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.885238    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-socket" pod "7456949c-c902-4940-b3e4-64fab197654b" (UID: "7456949c-c902-4940-b3e4-64fab197654b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:16:12.887142 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.887116    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-envoy" pod "7456949c-c902-4940-b3e4-64fab197654b" (UID: "7456949c-c902-4940-b3e4-64fab197654b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:16:12.891974 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.887630    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7456949c-c902-4940-b3e4-64fab197654b-istio-podinfo" pod "7456949c-c902-4940-b3e4-64fab197654b" (UID: "7456949c-c902-4940-b3e4-64fab197654b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:16:12.891974 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.887723    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-kube-api-access-6xlv7" pod "7456949c-c902-4940-b3e4-64fab197654b" (UID: "7456949c-c902-4940-b3e4-64fab197654b"). InnerVolumeSpecName "kube-api-access-6xlv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:16:12.891974 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.887908    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-istio-token" pod "7456949c-c902-4940-b3e4-64fab197654b" (UID: "7456949c-c902-4940-b3e4-64fab197654b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:16:12.985470 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.985434    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7456949c-c902-4940-b3e4-64fab197654b-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:16:12.985470 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.985464    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:16:12.985470 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.985474    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:16:12.985727 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.985483    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:16:12.985727 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.985492    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7456949c-c902-4940-b3e4-64fab197654b-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:16:12.985727 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.985501    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xlv7\" (UniqueName: \"kubernetes.io/projected/7456949c-c902-4940-b3e4-64fab197654b-kube-api-access-6xlv7\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:16:12.985727 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:12.985511    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7456949c-c902-4940-b3e4-64fab197654b-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:16:13.193730 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:13.193697    5224 generic.go:356] "Generic (PLEG): container finished" podID="7456949c-c902-4940-b3e4-64fab197654b" containerID="b16b9f23d4631b84f9d81e750f1f82ee57b8c62d3f87415048b057fdd259cc09" exitCode=0
Nov 02 21:16:13.193911 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:13.193756    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"
Nov 02 21:16:13.193911 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:13.193796    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" event={"ID":"7456949c-c902-4940-b3e4-64fab197654b","Type":"ContainerDied","Data":"b16b9f23d4631b84f9d81e750f1f82ee57b8c62d3f87415048b057fdd259cc09"}
Nov 02 21:16:13.193911 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:13.193853    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl" event={"ID":"7456949c-c902-4940-b3e4-64fab197654b","Type":"ContainerDied","Data":"e8672731b390848500b3bee1a5048f2b1726f4f285b950089b1c33eb19cb46a8"}
Nov 02 21:16:13.193911 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:13.193877    5224 scope.go:122] "RemoveContainer" containerID="b16b9f23d4631b84f9d81e750f1f82ee57b8c62d3f87415048b057fdd259cc09"
Nov 02 21:16:13.202545 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:13.202523    5224 scope.go:122] "RemoveContainer" containerID="b16b9f23d4631b84f9d81e750f1f82ee57b8c62d3f87415048b057fdd259cc09"
Nov 02 21:16:13.202845 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:16:13.202821    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b16b9f23d4631b84f9d81e750f1f82ee57b8c62d3f87415048b057fdd259cc09\": container with ID starting with b16b9f23d4631b84f9d81e750f1f82ee57b8c62d3f87415048b057fdd259cc09 not found: ID does not exist" containerID="b16b9f23d4631b84f9d81e750f1f82ee57b8c62d3f87415048b057fdd259cc09"
Nov 02 21:16:13.202951 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:13.202852    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b16b9f23d4631b84f9d81e750f1f82ee57b8c62d3f87415048b057fdd259cc09"} err="failed to get container status \"b16b9f23d4631b84f9d81e750f1f82ee57b8c62d3f87415048b057fdd259cc09\": rpc error: code = NotFound desc = could not find container \"b16b9f23d4631b84f9d81e750f1f82ee57b8c62d3f87415048b057fdd259cc09\": container with ID starting with b16b9f23d4631b84f9d81e750f1f82ee57b8c62d3f87415048b057fdd259cc09 not found: ID does not exist"
Nov 02 21:16:13.361840 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:13.361798    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"]
Nov 02 21:16:13.365230 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:13.365206    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d46vl"]
Nov 02 21:16:14.177536 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:14.177474    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:14.867167 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:14.867129    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:15.110263 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:15.110221    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:15.166184 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:15.166108    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7456949c-c902-4940-b3e4-64fab197654b" path="/var/lib/kubelet/pods/7456949c-c902-4940-b3e4-64fab197654b/volumes"
Nov 02 21:16:15.994232 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:15.994199    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:19.956285 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:19.956249    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:20.255632 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:20.255553    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:16:20.865964 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:20.865928    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:21.294638 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:21.294593    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:24.882484 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:24.882449    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:26.329151 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:26.329109    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:27.358450 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:16:27.358416    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:16:29.658679 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:29.658641    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:34.491209 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:34.491168    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:35.839636 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:35.839602    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:16:39.359320 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:39.359284    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:44.178393 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:44.178345    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:49.063034 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:49.062996    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:51.581995 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:51.581953    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:51.821070 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:51.821031    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:53.944701 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:53.944662    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:16:57.358268 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:16:57.358230    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:16:58.857171 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:16:58.857134    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:03.136178 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:03.136141    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:03.793193 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:03.793153    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:04.393738 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:04.393704    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:04.620299 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:04.620262    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:08.533330 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:08.533296    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:08.878070 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:08.877985    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:13.777609 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:13.777565    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:14.484658 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:14.484616    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:15.410546 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:15.410515    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:18.013140 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:18.013098    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:19.767472 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:19.767434    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:22.725908 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:22.725857    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:22.996876 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:22.996795    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:23.523380 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:23.523341    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:17:24.425086 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:24.425049    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:24.758235 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:24.758194    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:25.026038 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:25.026002    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:25.799001 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:25.798966    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:27.357613 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:17:27.357577    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:17:30.064090 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:30.064058    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:31.729883 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:31.729819    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:32.663800 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:32.663740    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:32.887338 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:32.887294    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:34.536728 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:34.536678    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:36.357803 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:36.357750    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:37.220267 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:37.220234    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:41.247814 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:41.247775    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:41.446927 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:41.446895    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:17:47.270220 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:47.270181    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:51.350543 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:51.350503    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:51.529609 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:51.529568    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:52.193299 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:52.193265    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:52.807016 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:52.806975    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:53.319373 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:53.319334    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:53.773908 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:53.773870    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:57.356982 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:17:57.356947    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:17:57.702125 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:57.702035    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:17:58.668974 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:58.668943    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"]
Nov 02 21:17:58.720084 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:58.720052    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"]
Nov 02 21:17:58.872319 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:17:58.872280    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:00.887702 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:00.887658    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:02.548297 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:02.548249    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:03.373074 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:03.373041    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:18:06.789672 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:06.789639    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:08.000515 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:08.000475    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:08.171077 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:08.171042    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:12.954643 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:12.954605    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:13.688668 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:13.688620    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs" podUID="c50013f5-4bb1-4b40-b62f-5b736b953ac0" containerName="main" containerID="cri-o://5d8d935c4ecc65a97b146839b952f08d74110cb2d836c33358326a5c0db685b1" gracePeriod=45
Nov 02 21:18:13.739213 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:13.739157    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk" podUID="41edc495-0352-461a-a80c-c4dd7358e0c3" containerName="main" containerID="cri-o://9a031f02161244c7504fb57b2a0b440444f6a722b8e8346fb5d98ae5ef3a2805" gracePeriod=45
Nov 02 21:18:14.006918 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.006892    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 21:18:14.009951 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.009931    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 21:18:14.066674 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.066641    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-dshm\") pod \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") "
Nov 02 21:18:14.066880 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.066688    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-tmp-dir\") pod \"41edc495-0352-461a-a80c-c4dd7358e0c3\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") "
Nov 02 21:18:14.066880 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.066731    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tls-certs\" (UniqueName: \"kubernetes.io/secret/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tls-certs\") pod \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") "
Nov 02 21:18:14.066880 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.066799    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41edc495-0352-461a-a80c-c4dd7358e0c3-kube-api-access-qlmb7\" (UniqueName: \"kubernetes.io/projected/41edc495-0352-461a-a80c-c4dd7358e0c3-kube-api-access-qlmb7\") pod \"41edc495-0352-461a-a80c-c4dd7358e0c3\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") "
Nov 02 21:18:14.066880 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.066838    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-model-cache\") pod \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") "
Nov 02 21:18:14.066880 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.066877    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-model-cache\") pod \"41edc495-0352-461a-a80c-c4dd7358e0c3\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") "
Nov 02 21:18:14.067203 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.066904    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-home\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-home\") pod \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") "
Nov 02 21:18:14.067203 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.066973    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41edc495-0352-461a-a80c-c4dd7358e0c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/41edc495-0352-461a-a80c-c4dd7358e0c3-tls-certs\") pod \"41edc495-0352-461a-a80c-c4dd7358e0c3\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") "
Nov 02 21:18:14.067203 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.067008    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-home\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-home\") pod \"41edc495-0352-461a-a80c-c4dd7358e0c3\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") "
Nov 02 21:18:14.067377 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.067253    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-home" pod "41edc495-0352-461a-a80c-c4dd7358e0c3" (UID: "41edc495-0352-461a-a80c-c4dd7358e0c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:18:14.068678 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.067590    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tmp-dir\") pod \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") "
Nov 02 21:18:14.068678 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.067749    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-model-cache" pod "41edc495-0352-461a-a80c-c4dd7358e0c3" (UID: "41edc495-0352-461a-a80c-c4dd7358e0c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:18:14.068678 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.067818    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-dshm\") pod \"41edc495-0352-461a-a80c-c4dd7358e0c3\" (UID: \"41edc495-0352-461a-a80c-c4dd7358e0c3\") "
Nov 02 21:18:14.068678 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.067884    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50013f5-4bb1-4b40-b62f-5b736b953ac0-kube-api-access-v9l8w\" (UniqueName: \"kubernetes.io/projected/c50013f5-4bb1-4b40-b62f-5b736b953ac0-kube-api-access-v9l8w\") pod \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\" (UID: \"c50013f5-4bb1-4b40-b62f-5b736b953ac0\") "
Nov 02 21:18:14.068678 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.068267    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-home" pod "c50013f5-4bb1-4b40-b62f-5b736b953ac0" (UID: "c50013f5-4bb1-4b40-b62f-5b736b953ac0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:18:14.068678 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.068321    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tmp-dir" pod "c50013f5-4bb1-4b40-b62f-5b736b953ac0" (UID: "c50013f5-4bb1-4b40-b62f-5b736b953ac0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:18:14.068678 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.068540    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:14.068678 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.068563    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:14.068678 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.068579    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:14.068678 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.068593    5224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-home\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:14.068678 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.068652    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-model-cache" pod "c50013f5-4bb1-4b40-b62f-5b736b953ac0" (UID: "c50013f5-4bb1-4b40-b62f-5b736b953ac0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:18:14.069635 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.069326    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-tmp-dir" pod "41edc495-0352-461a-a80c-c4dd7358e0c3" (UID: "41edc495-0352-461a-a80c-c4dd7358e0c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:18:14.071584 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.071542    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41edc495-0352-461a-a80c-c4dd7358e0c3-tls-certs" pod "41edc495-0352-461a-a80c-c4dd7358e0c3" (UID: "41edc495-0352-461a-a80c-c4dd7358e0c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:18:14.071584 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.071551    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-dshm" pod "41edc495-0352-461a-a80c-c4dd7358e0c3" (UID: "41edc495-0352-461a-a80c-c4dd7358e0c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:18:14.071584 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.071570    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-dshm" pod "c50013f5-4bb1-4b40-b62f-5b736b953ac0" (UID: "c50013f5-4bb1-4b40-b62f-5b736b953ac0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:18:14.071809 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.071585    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41edc495-0352-461a-a80c-c4dd7358e0c3-kube-api-access-qlmb7" pod "41edc495-0352-461a-a80c-c4dd7358e0c3" (UID: "41edc495-0352-461a-a80c-c4dd7358e0c3"). InnerVolumeSpecName "kube-api-access-qlmb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:18:14.071809 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.071558    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tls-certs" pod "c50013f5-4bb1-4b40-b62f-5b736b953ac0" (UID: "c50013f5-4bb1-4b40-b62f-5b736b953ac0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:18:14.071809 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.071757    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50013f5-4bb1-4b40-b62f-5b736b953ac0-kube-api-access-v9l8w" pod "c50013f5-4bb1-4b40-b62f-5b736b953ac0" (UID: "c50013f5-4bb1-4b40-b62f-5b736b953ac0"). InnerVolumeSpecName "kube-api-access-v9l8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:18:14.170239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.169998    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9l8w\" (UniqueName: \"kubernetes.io/projected/c50013f5-4bb1-4b40-b62f-5b736b953ac0-kube-api-access-v9l8w\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:14.170239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.170033    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:14.170239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.170049    5224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-tmp-dir\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:14.170239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.170062    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c50013f5-4bb1-4b40-b62f-5b736b953ac0-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:14.170239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.170075    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlmb7\" (UniqueName: \"kubernetes.io/projected/41edc495-0352-461a-a80c-c4dd7358e0c3-kube-api-access-qlmb7\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:14.170239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.170089    5224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c50013f5-4bb1-4b40-b62f-5b736b953ac0-model-cache\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:14.170239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.170103    5224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41edc495-0352-461a-a80c-c4dd7358e0c3-tls-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:14.170239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.170118    5224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41edc495-0352-461a-a80c-c4dd7358e0c3-dshm\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:14.763555 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.763518    5224 generic.go:356] "Generic (PLEG): container finished" podID="41edc495-0352-461a-a80c-c4dd7358e0c3" containerID="9a031f02161244c7504fb57b2a0b440444f6a722b8e8346fb5d98ae5ef3a2805" exitCode=0
Nov 02 21:18:14.763731 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.763566    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"
Nov 02 21:18:14.763731 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.763604    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk" event={"ID":"41edc495-0352-461a-a80c-c4dd7358e0c3","Type":"ContainerDied","Data":"9a031f02161244c7504fb57b2a0b440444f6a722b8e8346fb5d98ae5ef3a2805"}
Nov 02 21:18:14.763731 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.763648    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk" event={"ID":"41edc495-0352-461a-a80c-c4dd7358e0c3","Type":"ContainerDied","Data":"27e40820862f773bc32e0ff08b54b517ef1aff9e9f86b0fd4034fbaa81e29ac0"}
Nov 02 21:18:14.763731 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.763668    5224 scope.go:122] "RemoveContainer" containerID="9a031f02161244c7504fb57b2a0b440444f6a722b8e8346fb5d98ae5ef3a2805"
Nov 02 21:18:14.765025 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.764969    5224 generic.go:356] "Generic (PLEG): container finished" podID="c50013f5-4bb1-4b40-b62f-5b736b953ac0" containerID="5d8d935c4ecc65a97b146839b952f08d74110cb2d836c33358326a5c0db685b1" exitCode=0
Nov 02 21:18:14.765025 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.765021    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"
Nov 02 21:18:14.765127 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.765058    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs" event={"ID":"c50013f5-4bb1-4b40-b62f-5b736b953ac0","Type":"ContainerDied","Data":"5d8d935c4ecc65a97b146839b952f08d74110cb2d836c33358326a5c0db685b1"}
Nov 02 21:18:14.765127 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.765084    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs" event={"ID":"c50013f5-4bb1-4b40-b62f-5b736b953ac0","Type":"ContainerDied","Data":"36697a7b3f560d98da048b70e0aca8ebdc113f4b069d7076e2ace949b6dc20c9"}
Nov 02 21:18:14.776910 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.776869    5224 scope.go:122] "RemoveContainer" containerID="9a031f02161244c7504fb57b2a0b440444f6a722b8e8346fb5d98ae5ef3a2805"
Nov 02 21:18:14.777164 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:18:14.777142    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a031f02161244c7504fb57b2a0b440444f6a722b8e8346fb5d98ae5ef3a2805\": container with ID starting with 9a031f02161244c7504fb57b2a0b440444f6a722b8e8346fb5d98ae5ef3a2805 not found: ID does not exist" containerID="9a031f02161244c7504fb57b2a0b440444f6a722b8e8346fb5d98ae5ef3a2805"
Nov 02 21:18:14.777250 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.777178    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a031f02161244c7504fb57b2a0b440444f6a722b8e8346fb5d98ae5ef3a2805"} err="failed to get container status \"9a031f02161244c7504fb57b2a0b440444f6a722b8e8346fb5d98ae5ef3a2805\": rpc error: code = NotFound desc = could not find container \"9a031f02161244c7504fb57b2a0b440444f6a722b8e8346fb5d98ae5ef3a2805\": container with ID starting with 9a031f02161244c7504fb57b2a0b440444f6a722b8e8346fb5d98ae5ef3a2805 not found: ID does not exist"
Nov 02 21:18:14.777250 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.777203    5224 scope.go:122] "RemoveContainer" containerID="5d8d935c4ecc65a97b146839b952f08d74110cb2d836c33358326a5c0db685b1"
Nov 02 21:18:14.784586 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.784566    5224 scope.go:122] "RemoveContainer" containerID="5d8d935c4ecc65a97b146839b952f08d74110cb2d836c33358326a5c0db685b1"
Nov 02 21:18:14.784856 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:18:14.784837    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8d935c4ecc65a97b146839b952f08d74110cb2d836c33358326a5c0db685b1\": container with ID starting with 5d8d935c4ecc65a97b146839b952f08d74110cb2d836c33358326a5c0db685b1 not found: ID does not exist" containerID="5d8d935c4ecc65a97b146839b952f08d74110cb2d836c33358326a5c0db685b1"
Nov 02 21:18:14.784910 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.784864    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8d935c4ecc65a97b146839b952f08d74110cb2d836c33358326a5c0db685b1"} err="failed to get container status \"5d8d935c4ecc65a97b146839b952f08d74110cb2d836c33358326a5c0db685b1\": rpc error: code = NotFound desc = could not find container \"5d8d935c4ecc65a97b146839b952f08d74110cb2d836c33358326a5c0db685b1\": container with ID starting with 5d8d935c4ecc65a97b146839b952f08d74110cb2d836c33358326a5c0db685b1 not found: ID does not exist"
Nov 02 21:18:14.820824 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.820797    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:14.930248 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.930214    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"]
Nov 02 21:18:14.981672 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:14.981603    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/distinct-2-w0-14d090-kserve-6b58bcf874-n22gs"]
Nov 02 21:18:15.094984 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:15.094950    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"]
Nov 02 21:18:15.097886 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:15.097863    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/simulator-w0-14d090-kserve-598df86587-9n6gk"]
Nov 02 21:18:15.166234 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:15.166198    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41edc495-0352-461a-a80c-c4dd7358e0c3" path="/var/lib/kubelet/pods/41edc495-0352-461a-a80c-c4dd7358e0c3/volumes"
Nov 02 21:18:15.166672 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:15.166651    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50013f5-4bb1-4b40-b62f-5b736b953ac0" path="/var/lib/kubelet/pods/c50013f5-4bb1-4b40-b62f-5b736b953ac0/volumes"
Nov 02 21:18:15.520914 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:15.520879    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:17.750782 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:17.750731    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:17.981313 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:17.981278    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:18.470197 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:18.470151    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:18.991820 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:18.991761    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:19.689650 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.689613    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"]
Nov 02 21:18:19.689978 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.689964    5224 cpu_manager.go:414] "containerMap: removing container" podUID="c50013f5-4bb1-4b40-b62f-5b736b953ac0" containerName="main"
Nov 02 21:18:19.690033 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.689979    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50013f5-4bb1-4b40-b62f-5b736b953ac0" containerName="main"
Nov 02 21:18:19.690033 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690004    5224 cpu_manager.go:414] "containerMap: removing container" podUID="7456949c-c902-4940-b3e4-64fab197654b" containerName="istio-proxy"
Nov 02 21:18:19.690033 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690010    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7456949c-c902-4940-b3e4-64fab197654b" containerName="istio-proxy"
Nov 02 21:18:19.690033 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690018    5224 cpu_manager.go:414] "containerMap: removing container" podUID="a18abf8b-0c6a-4d86-9a76-55aaa395298e" containerName="cleanup"
Nov 02 21:18:19.690033 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690023    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18abf8b-0c6a-4d86-9a76-55aaa395298e" containerName="cleanup"
Nov 02 21:18:19.690033 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690035    5224 cpu_manager.go:414] "containerMap: removing container" podUID="08fbda88-2e0b-4119-8fb4-17480f431ed9" containerName="revoke-keys"
Nov 02 21:18:19.690245 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690040    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fbda88-2e0b-4119-8fb4-17480f431ed9" containerName="revoke-keys"
Nov 02 21:18:19.690245 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690046    5224 cpu_manager.go:414] "containerMap: removing container" podUID="41edc495-0352-461a-a80c-c4dd7358e0c3" containerName="main"
Nov 02 21:18:19.690245 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690053    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41edc495-0352-461a-a80c-c4dd7358e0c3" containerName="main"
Nov 02 21:18:19.690245 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690115    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7456949c-c902-4940-b3e4-64fab197654b" containerName="istio-proxy"
Nov 02 21:18:19.690245 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690121    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18abf8b-0c6a-4d86-9a76-55aaa395298e" containerName="cleanup"
Nov 02 21:18:19.690245 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690128    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59f8676-3618-4d0b-a026-6461cb0c195f" containerName="cleanup"
Nov 02 21:18:19.690245 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690134    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fbda88-2e0b-4119-8fb4-17480f431ed9" containerName="revoke-keys"
Nov 02 21:18:19.690245 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690139    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="41edc495-0352-461a-a80c-c4dd7358e0c3" containerName="main"
Nov 02 21:18:19.690245 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.690147    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50013f5-4bb1-4b40-b62f-5b736b953ac0" containerName="main"
Nov 02 21:18:19.694817 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.694794    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.719128 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.719092    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be06ceb-11fc-4de1-85ab-a048f548feeb-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.719489 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.719140    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.719489 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.719223    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.719489 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.719287    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h68cs\" (UniqueName: \"kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-kube-api-access-h68cs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.719489 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.719305    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.719489 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.719331    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.719489 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.719348    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.719489 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.719453    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.719883 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.719517    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.748009 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.747972    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-14d090-openshift-default-dockercfg-5wl9l\""
Nov 02 21:18:19.800780 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.800729    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"]
Nov 02 21:18:19.820558 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.820528    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.820750 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.820589    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.820750 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.820615    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be06ceb-11fc-4de1-85ab-a048f548feeb-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.820750 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.820646    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.820750 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.820667    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.820750 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.820712    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h68cs\" (UniqueName: \"kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-kube-api-access-h68cs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.820750 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.820741    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.821124 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.820791    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.821124 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.820833    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.821239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.821142    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.821239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.821229    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.821350 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.821302    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.821408 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.821391    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.821470 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.821448    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be06ceb-11fc-4de1-85ab-a048f548feeb-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.823046 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.823018    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.823240 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.823225    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.878710 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.878674    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:19.878882 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:19.878742    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h68cs\" (UniqueName: \"kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-kube-api-access-h68cs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:20.007922 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:20.007840    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:20.419563 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:18:20.419530    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be06ceb_11fc_4de1_85ab_a048f548feeb.slice/crio-95a57c5595774b18ac833f428f768b9090e755479cc9dd038624965abdbcefae WatchSource:0}: Error finding container 95a57c5595774b18ac833f428f768b9090e755479cc9dd038624965abdbcefae: Status 404 returned error can't find the container with id 95a57c5595774b18ac833f428f768b9090e755479cc9dd038624965abdbcefae
Nov 02 21:18:20.421705 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:20.421671    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:18:20.421808 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:20.421738    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:18:20.421808 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:20.421794    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:18:20.421972 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:20.421952    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"]
Nov 02 21:18:20.799913 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:20.799874    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq" event={"ID":"1be06ceb-11fc-4de1-85ab-a048f548feeb","Type":"ContainerStarted","Data":"506a1d8c15b2cb4d9a14c31727a5d456214c8e395f325bc2f9f044cc1bfc653f"}
Nov 02 21:18:20.800104 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:20.799934    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq" event={"ID":"1be06ceb-11fc-4de1-85ab-a048f548feeb","Type":"ContainerStarted","Data":"95a57c5595774b18ac833f428f768b9090e755479cc9dd038624965abdbcefae"}
Nov 02 21:18:20.930728 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:20.930674    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq" podStartSLOduration=1.9306594879999999 podStartE2EDuration="1.930659488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:20.928934941 +0000 UTC m=+5381.926623550" watchObservedRunningTime="2026-11-02 21:18:20.930659488 +0000 UTC m=+5381.928348097"
Nov 02 21:18:21.008506 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:21.008470    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:21.010105 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:21.010076    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 02 21:18:21.010227 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:21.010127    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq" podUID="1be06ceb-11fc-4de1-85ab-a048f548feeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 02 21:18:22.008652 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:22.008617    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 02 21:18:22.009041 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:22.008669    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq" podUID="1be06ceb-11fc-4de1-85ab-a048f548feeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 02 21:18:22.172791 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:22.172741    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:23.008231 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:23.008197    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 02 21:18:23.008429 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:23.008252    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq" podUID="1be06ceb-11fc-4de1-85ab-a048f548feeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 02 21:18:24.009112 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:24.009073    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 02 21:18:24.009488 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:24.009134    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq" podUID="1be06ceb-11fc-4de1-85ab-a048f548feeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 02 21:18:24.123067 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:24.123031    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:24.303474 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:24.303438    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:24.914685 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:24.914644    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:25.009029 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:25.008998    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 02 21:18:25.009234 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:25.009057    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq" podUID="1be06ceb-11fc-4de1-85ab-a048f548feeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 02 21:18:25.199285 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:25.199189    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:25.352793 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:25.350263    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:18:25.721361 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:25.721325    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:26.008612 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:26.008538    5224 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 02 21:18:26.008612 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:26.008588    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq" podUID="1be06ceb-11fc-4de1-85ab-a048f548feeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 02 21:18:26.147675 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:26.147628    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"]
Nov 02 21:18:26.148133 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:26.147833    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq" podUID="1be06ceb-11fc-4de1-85ab-a048f548feeb" containerName="istio-proxy" containerID="cri-o://506a1d8c15b2cb4d9a14c31727a5d456214c8e395f325bc2f9f044cc1bfc653f" gracePeriod=30
Nov 02 21:18:27.356878 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:18:27.356841    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:18:27.494338 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:27.494300    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:29.399669 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:29.399638    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2ptm5_1f554e67-b005-4058-9728-e047ee0f014b/limitador/0.log"
Nov 02 21:18:29.630606 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:29.630569    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rkw4k_69235044-5904-4cad-93db-315e6e36f8fd/manager/0.log"
Nov 02 21:18:31.062595 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:31.062564    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2vzvt_cb2304f9-c44a-44a4-a0b5-6029ffd68eec/manager/0.log"
Nov 02 21:18:31.551820 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:31.551789    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8z8gw_e49ef931-b6e2-4394-80f3-881252323f43/manager/0.log"
Nov 02 21:18:32.161753 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:32.161723    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:32.377004 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:32.376973    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:33.812378 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.812350    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:33.860497 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.860466    5224 generic.go:356] "Generic (PLEG): container finished" podID="1be06ceb-11fc-4de1-85ab-a048f548feeb" containerID="506a1d8c15b2cb4d9a14c31727a5d456214c8e395f325bc2f9f044cc1bfc653f" exitCode=0
Nov 02 21:18:33.860652 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.860509    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"
Nov 02 21:18:33.860652 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.860551    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq" event={"ID":"1be06ceb-11fc-4de1-85ab-a048f548feeb","Type":"ContainerDied","Data":"506a1d8c15b2cb4d9a14c31727a5d456214c8e395f325bc2f9f044cc1bfc653f"}
Nov 02 21:18:33.860652 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.860602    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq" event={"ID":"1be06ceb-11fc-4de1-85ab-a048f548feeb","Type":"ContainerDied","Data":"95a57c5595774b18ac833f428f768b9090e755479cc9dd038624965abdbcefae"}
Nov 02 21:18:33.860652 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.860625    5224 scope.go:122] "RemoveContainer" containerID="506a1d8c15b2cb4d9a14c31727a5d456214c8e395f325bc2f9f044cc1bfc653f"
Nov 02 21:18:33.869132 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.869112    5224 scope.go:122] "RemoveContainer" containerID="506a1d8c15b2cb4d9a14c31727a5d456214c8e395f325bc2f9f044cc1bfc653f"
Nov 02 21:18:33.869381 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:18:33.869363    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506a1d8c15b2cb4d9a14c31727a5d456214c8e395f325bc2f9f044cc1bfc653f\": container with ID starting with 506a1d8c15b2cb4d9a14c31727a5d456214c8e395f325bc2f9f044cc1bfc653f not found: ID does not exist" containerID="506a1d8c15b2cb4d9a14c31727a5d456214c8e395f325bc2f9f044cc1bfc653f"
Nov 02 21:18:33.869457 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.869392    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506a1d8c15b2cb4d9a14c31727a5d456214c8e395f325bc2f9f044cc1bfc653f"} err="failed to get container status \"506a1d8c15b2cb4d9a14c31727a5d456214c8e395f325bc2f9f044cc1bfc653f\": rpc error: code = NotFound desc = could not find container \"506a1d8c15b2cb4d9a14c31727a5d456214c8e395f325bc2f9f044cc1bfc653f\": container with ID starting with 506a1d8c15b2cb4d9a14c31727a5d456214c8e395f325bc2f9f044cc1bfc653f not found: ID does not exist"
Nov 02 21:18:33.870623 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.870607    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-certs\") pod \"1be06ceb-11fc-4de1-85ab-a048f548feeb\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") "
Nov 02 21:18:33.870677 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.870634    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-envoy\") pod \"1be06ceb-11fc-4de1-85ab-a048f548feeb\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") "
Nov 02 21:18:33.870677 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.870660    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-token\" (UniqueName: \"kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-token\") pod \"1be06ceb-11fc-4de1-85ab-a048f548feeb\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") "
Nov 02 21:18:33.870756 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.870683    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be06ceb-11fc-4de1-85ab-a048f548feeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be06ceb-11fc-4de1-85ab-a048f548feeb-istiod-ca-cert\") pod \"1be06ceb-11fc-4de1-85ab-a048f548feeb\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") "
Nov 02 21:18:33.870756 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.870705    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-credential-socket\") pod \"1be06ceb-11fc-4de1-85ab-a048f548feeb\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") "
Nov 02 21:18:33.870756 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.870752    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-socket\") pod \"1be06ceb-11fc-4de1-85ab-a048f548feeb\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") "
Nov 02 21:18:33.870941 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.870811    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-kube-api-access-h68cs\" (UniqueName: \"kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-kube-api-access-h68cs\") pod \"1be06ceb-11fc-4de1-85ab-a048f548feeb\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") "
Nov 02 21:18:33.870941 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.870831    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-podinfo\") pod \"1be06ceb-11fc-4de1-85ab-a048f548feeb\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") "
Nov 02 21:18:33.870941 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.870849    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-data\") pod \"1be06ceb-11fc-4de1-85ab-a048f548feeb\" (UID: \"1be06ceb-11fc-4de1-85ab-a048f548feeb\") "
Nov 02 21:18:33.871112 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.871074    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-credential-socket" pod "1be06ceb-11fc-4de1-85ab-a048f548feeb" (UID: "1be06ceb-11fc-4de1-85ab-a048f548feeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:18:33.871112 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.871080    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be06ceb-11fc-4de1-85ab-a048f548feeb-istiod-ca-cert" pod "1be06ceb-11fc-4de1-85ab-a048f548feeb" (UID: "1be06ceb-11fc-4de1-85ab-a048f548feeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:18:33.871234 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.871198    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be06ceb-11fc-4de1-85ab-a048f548feeb-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:33.871291 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.871255    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:33.871291 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.871257    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-certs" pod "1be06ceb-11fc-4de1-85ab-a048f548feeb" (UID: "1be06ceb-11fc-4de1-85ab-a048f548feeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:18:33.871412 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.871288    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-data" pod "1be06ceb-11fc-4de1-85ab-a048f548feeb" (UID: "1be06ceb-11fc-4de1-85ab-a048f548feeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:18:33.871412 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.871320    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-socket" pod "1be06ceb-11fc-4de1-85ab-a048f548feeb" (UID: "1be06ceb-11fc-4de1-85ab-a048f548feeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:18:33.873194 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.873169    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-podinfo" pod "1be06ceb-11fc-4de1-85ab-a048f548feeb" (UID: "1be06ceb-11fc-4de1-85ab-a048f548feeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:18:33.873286 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.873198    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-kube-api-access-h68cs" pod "1be06ceb-11fc-4de1-85ab-a048f548feeb" (UID: "1be06ceb-11fc-4de1-85ab-a048f548feeb"). InnerVolumeSpecName "kube-api-access-h68cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:18:33.873286 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.873210    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-token" pod "1be06ceb-11fc-4de1-85ab-a048f548feeb" (UID: "1be06ceb-11fc-4de1-85ab-a048f548feeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:18:33.873286 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.873264    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-envoy" pod "1be06ceb-11fc-4de1-85ab-a048f548feeb" (UID: "1be06ceb-11fc-4de1-85ab-a048f548feeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:18:33.972116 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.972046    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:33.972116 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.972069    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:33.972116 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.972079    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:33.972116 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.972088    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:33.972116 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.972097    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h68cs\" (UniqueName: \"kubernetes.io/projected/1be06ceb-11fc-4de1-85ab-a048f548feeb-kube-api-access-h68cs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:33.972116 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.972107    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:33.972116 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:33.972115    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be06ceb-11fc-4de1-85ab-a048f548feeb-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:34.283153 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:34.283113    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"]
Nov 02 21:18:34.286955 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:34.286935    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-qp2tq"]
Nov 02 21:18:35.166166 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:35.166129    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be06ceb-11fc-4de1-85ab-a048f548feeb" path="/var/lib/kubelet/pods/1be06ceb-11fc-4de1-85ab-a048f548feeb/volumes"
Nov 02 21:18:37.532109 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:37.532070    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:38.115362 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:38.115327    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:38.432503 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:38.432418    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:41.302958 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:41.302904    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:41.506877 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:41.506800    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:41.699846 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:41.699804    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:44.432971 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:44.432938    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:44.630196 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:44.630161    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:47.702632 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:47.702575    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:47.922258 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:47.922224    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:52.745990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:52.745877    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:18:56.266456 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:56.266420    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:18:57.357660 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:18:57.357620    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:18:57.989451 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:18:57.989411    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:02.681048 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:02.681013    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:07.992842 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:07.992809    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:13.332471 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:13.332422    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:15.174410 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:15.174365    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:15.840080 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:15.840044    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:18.188915 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:18.188876    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:23.081173 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:23.081130    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:26.336970 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:26.336924    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:27.357068 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:19:27.357032    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:19:28.016342 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:28.016303    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:29.495957 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:29.495924    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:29.698059 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:29.698025    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:32.757178 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:32.757137    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:37.050203 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:37.050165    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:38.593962 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:38.593915    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:38.877685 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:38.877598    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:41.241551 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:41.241507    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:19:43.881542 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:19:43.881508    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:19:57.357937 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:19:57.357899    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:20:11.395379 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.395331    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"]
Nov 02 21:20:11.397932 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.395695    5224 cpu_manager.go:414] "containerMap: removing container" podUID="1be06ceb-11fc-4de1-85ab-a048f548feeb" containerName="istio-proxy"
Nov 02 21:20:11.397932 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.395706    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be06ceb-11fc-4de1-85ab-a048f548feeb" containerName="istio-proxy"
Nov 02 21:20:11.397932 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.395797    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be06ceb-11fc-4de1-85ab-a048f548feeb" containerName="istio-proxy"
Nov 02 21:20:11.398753 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.398735    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"
Nov 02 21:20:11.454802 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.454757    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"]
Nov 02 21:20:11.492896 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.492855    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt6nj\" (UniqueName: \"kubernetes.io/projected/1e66d2a8-994e-467c-ac11-47b3b34db7c6-kube-api-access-pt6nj\") pod \"maas-api-revoke-sub-85237e45-zd572\" (UID: \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"
Nov 02 21:20:11.493077 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.492911    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-zd572\" (UID: \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"
Nov 02 21:20:11.493077 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.492979    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-zd572\" (UID: \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"
Nov 02 21:20:11.504267 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.504238    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:20:11.504413 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.504277    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:20:11.504413 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.504282    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\""
Nov 02 21:20:11.504413 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.504328    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:20:11.594094 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.594061    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6nj\" (UniqueName: \"kubernetes.io/projected/1e66d2a8-994e-467c-ac11-47b3b34db7c6-kube-api-access-pt6nj\") pod \"maas-api-revoke-sub-85237e45-zd572\" (UID: \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"
Nov 02 21:20:11.594277 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.594103    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-zd572\" (UID: \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"
Nov 02 21:20:11.594277 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.594126    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-zd572\" (UID: \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"
Nov 02 21:20:11.594723 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.594705    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-zd572\" (UID: \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"
Nov 02 21:20:11.596465 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.596445    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-zd572\" (UID: \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"
Nov 02 21:20:11.654428 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.654342    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6nj\" (UniqueName: \"kubernetes.io/projected/1e66d2a8-994e-467c-ac11-47b3b34db7c6-kube-api-access-pt6nj\") pod \"maas-api-revoke-sub-85237e45-zd572\" (UID: \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"
Nov 02 21:20:11.711341 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:11.711291    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"
Nov 02 21:20:12.076596 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:12.076566    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"]
Nov 02 21:20:12.305032 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:12.305001    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572" event={"ID":"1e66d2a8-994e-467c-ac11-47b3b34db7c6","Type":"ContainerStarted","Data":"5b0339a5d0909ebc23e4b5798d3a0019394b8865eedb1e5e6dd0339a082d3173"}
Nov 02 21:20:13.311508 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:13.311471    5224 generic.go:356] "Generic (PLEG): container finished" podID="1e66d2a8-994e-467c-ac11-47b3b34db7c6" containerID="5e8bae27d7e45cf7eb0cb6fb5406dd09f8e41a54bbbaa862404fb4db41ef2df7" exitCode=0
Nov 02 21:20:13.311956 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:13.311546    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572" event={"ID":"1e66d2a8-994e-467c-ac11-47b3b34db7c6","Type":"ContainerDied","Data":"5e8bae27d7e45cf7eb0cb6fb5406dd09f8e41a54bbbaa862404fb4db41ef2df7"}
Nov 02 21:20:14.438889 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:14.438860    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"
Nov 02 21:20:14.524490 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:14.524447    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e66d2a8-994e-467c-ac11-47b3b34db7c6-kube-api-access-pt6nj\" (UniqueName: \"kubernetes.io/projected/1e66d2a8-994e-467c-ac11-47b3b34db7c6-kube-api-access-pt6nj\") pod \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\" (UID: \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\") "
Nov 02 21:20:14.524683 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:14.524547    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-xks-ca\") pod \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\" (UID: \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\") "
Nov 02 21:20:14.524683 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:14.524599    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-openshift-ca\") pod \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\" (UID: \"1e66d2a8-994e-467c-ac11-47b3b34db7c6\") "
Nov 02 21:20:14.525040 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:14.525016    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-openshift-ca" pod "1e66d2a8-994e-467c-ac11-47b3b34db7c6" (UID: "1e66d2a8-994e-467c-ac11-47b3b34db7c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:20:14.526709 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:14.526683    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-xks-ca" pod "1e66d2a8-994e-467c-ac11-47b3b34db7c6" (UID: "1e66d2a8-994e-467c-ac11-47b3b34db7c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:20:14.526835 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:14.526722    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e66d2a8-994e-467c-ac11-47b3b34db7c6-kube-api-access-pt6nj" pod "1e66d2a8-994e-467c-ac11-47b3b34db7c6" (UID: "1e66d2a8-994e-467c-ac11-47b3b34db7c6"). InnerVolumeSpecName "kube-api-access-pt6nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:20:14.625629 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:14.625519    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt6nj\" (UniqueName: \"kubernetes.io/projected/1e66d2a8-994e-467c-ac11-47b3b34db7c6-kube-api-access-pt6nj\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:14.625629 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:14.625595    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:14.625629 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:14.625607    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e66d2a8-994e-467c-ac11-47b3b34db7c6-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:15.322730 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:15.322693    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572" event={"ID":"1e66d2a8-994e-467c-ac11-47b3b34db7c6","Type":"ContainerDied","Data":"5b0339a5d0909ebc23e4b5798d3a0019394b8865eedb1e5e6dd0339a082d3173"}
Nov 02 21:20:15.322730 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:15.322727    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"
Nov 02 21:20:15.322981 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:15.322755    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b0339a5d0909ebc23e4b5798d3a0019394b8865eedb1e5e6dd0339a082d3173"
Nov 02 21:20:15.547352 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:15.547323    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:20:16.383720 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:16.383683    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:20:23.756380 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:23.756279    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:20:26.039758 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:26.039722    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:20:27.359344 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:20:27.359307    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:20:34.046612 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:34.046571    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"]
Nov 02 21:20:34.098847 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:34.098804    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c11b4cae-pxsc4"]
Nov 02 21:20:35.165651 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:35.165603    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08fbda88-2e0b-4119-8fb4-17480f431ed9" path="/var/lib/kubelet/pods/08fbda88-2e0b-4119-8fb4-17480f431ed9/volumes"
Nov 02 21:20:46.570890 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:46.570851    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:20:52.361426 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:52.361386    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:20:57.358666 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:20:57.358631    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:20:58.511662 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.511629    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"]
Nov 02 21:20:58.512083 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.511996    5224 cpu_manager.go:414] "containerMap: removing container" podUID="1e66d2a8-994e-467c-ac11-47b3b34db7c6" containerName="revoke-keys"
Nov 02 21:20:58.512083 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.512008    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e66d2a8-994e-467c-ac11-47b3b34db7c6" containerName="revoke-keys"
Nov 02 21:20:58.512166 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.512105    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e66d2a8-994e-467c-ac11-47b3b34db7c6" containerName="revoke-keys"
Nov 02 21:20:58.514882 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.514864    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"
Nov 02 21:20:58.572031 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.571999    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"]
Nov 02 21:20:58.613075 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.613038    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b60203-3cb9-4133-813a-30a47a224770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-bj7zg\" (UID: \"48b60203-3cb9-4133-813a-30a47a224770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"
Nov 02 21:20:58.613249 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.613099    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l728h\" (UniqueName: \"kubernetes.io/projected/48b60203-3cb9-4133-813a-30a47a224770-kube-api-access-l728h\") pod \"maas-api-revoke-sub-f904bff9-bj7zg\" (UID: \"48b60203-3cb9-4133-813a-30a47a224770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"
Nov 02 21:20:58.613249 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.613163    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b60203-3cb9-4133-813a-30a47a224770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-bj7zg\" (UID: \"48b60203-3cb9-4133-813a-30a47a224770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"
Nov 02 21:20:58.621435 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.621403    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:20:58.621435 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.621404    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:20:58.621435 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.621434    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\""
Nov 02 21:20:58.621651 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.621488    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:20:58.713638 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.713598    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l728h\" (UniqueName: \"kubernetes.io/projected/48b60203-3cb9-4133-813a-30a47a224770-kube-api-access-l728h\") pod \"maas-api-revoke-sub-f904bff9-bj7zg\" (UID: \"48b60203-3cb9-4133-813a-30a47a224770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"
Nov 02 21:20:58.713638 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.713640    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b60203-3cb9-4133-813a-30a47a224770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-bj7zg\" (UID: \"48b60203-3cb9-4133-813a-30a47a224770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"
Nov 02 21:20:58.713966 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.713697    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b60203-3cb9-4133-813a-30a47a224770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-bj7zg\" (UID: \"48b60203-3cb9-4133-813a-30a47a224770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"
Nov 02 21:20:58.714417 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.714397    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b60203-3cb9-4133-813a-30a47a224770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-bj7zg\" (UID: \"48b60203-3cb9-4133-813a-30a47a224770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"
Nov 02 21:20:58.716004 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.715978    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b60203-3cb9-4133-813a-30a47a224770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-bj7zg\" (UID: \"48b60203-3cb9-4133-813a-30a47a224770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"
Nov 02 21:20:58.772160 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.772130    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l728h\" (UniqueName: \"kubernetes.io/projected/48b60203-3cb9-4133-813a-30a47a224770-kube-api-access-l728h\") pod \"maas-api-revoke-sub-f904bff9-bj7zg\" (UID: \"48b60203-3cb9-4133-813a-30a47a224770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"
Nov 02 21:20:58.827166 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:58.827124    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"
Nov 02 21:20:59.189862 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:20:59.189815    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b60203_3cb9_4133_813a_30a47a224770.slice/crio-d3690653ed9a9c4382b9fad08990487bf83c3b357823caac03524b3f4de68d0e WatchSource:0}: Error finding container d3690653ed9a9c4382b9fad08990487bf83c3b357823caac03524b3f4de68d0e: Status 404 returned error can't find the container with id d3690653ed9a9c4382b9fad08990487bf83c3b357823caac03524b3f4de68d0e
Nov 02 21:20:59.191883 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:59.191856    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"]
Nov 02 21:20:59.524857 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:59.524820    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg" event={"ID":"48b60203-3cb9-4133-813a-30a47a224770","Type":"ContainerStarted","Data":"fa6402353d3490a392ed5aa5f6e0af1e6cfc9d4e5cf3379587506959a7cc13f8"}
Nov 02 21:20:59.525288 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:59.524884    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg" event={"ID":"48b60203-3cb9-4133-813a-30a47a224770","Type":"ContainerStarted","Data":"d3690653ed9a9c4382b9fad08990487bf83c3b357823caac03524b3f4de68d0e"}
Nov 02 21:20:59.640563 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:20:59.640512    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg" podStartSLOduration=1.640498212 podStartE2EDuration="1.640498212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:59.639650407 +0000 UTC m=+5540.637339016" watchObservedRunningTime="2026-11-02 21:20:59.640498212 +0000 UTC m=+5540.638186822"
Nov 02 21:21:00.530932 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:00.530897    5224 generic.go:356] "Generic (PLEG): container finished" podID="48b60203-3cb9-4133-813a-30a47a224770" containerID="fa6402353d3490a392ed5aa5f6e0af1e6cfc9d4e5cf3379587506959a7cc13f8" exitCode=0
Nov 02 21:21:00.531326 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:00.530984    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg" event={"ID":"48b60203-3cb9-4133-813a-30a47a224770","Type":"ContainerDied","Data":"fa6402353d3490a392ed5aa5f6e0af1e6cfc9d4e5cf3379587506959a7cc13f8"}
Nov 02 21:21:01.662434 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:01.662409    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"
Nov 02 21:21:01.737055 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:01.737015    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48b60203-3cb9-4133-813a-30a47a224770-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b60203-3cb9-4133-813a-30a47a224770-maas-api-xks-ca\") pod \"48b60203-3cb9-4133-813a-30a47a224770\" (UID: \"48b60203-3cb9-4133-813a-30a47a224770\") "
Nov 02 21:21:01.737055 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:01.737062    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b60203-3cb9-4133-813a-30a47a224770-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b60203-3cb9-4133-813a-30a47a224770-maas-api-openshift-ca\") pod \"48b60203-3cb9-4133-813a-30a47a224770\" (UID: \"48b60203-3cb9-4133-813a-30a47a224770\") "
Nov 02 21:21:01.737334 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:01.737086    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b60203-3cb9-4133-813a-30a47a224770-kube-api-access-l728h\" (UniqueName: \"kubernetes.io/projected/48b60203-3cb9-4133-813a-30a47a224770-kube-api-access-l728h\") pod \"48b60203-3cb9-4133-813a-30a47a224770\" (UID: \"48b60203-3cb9-4133-813a-30a47a224770\") "
Nov 02 21:21:01.737460 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:01.737435    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b60203-3cb9-4133-813a-30a47a224770-maas-api-openshift-ca" pod "48b60203-3cb9-4133-813a-30a47a224770" (UID: "48b60203-3cb9-4133-813a-30a47a224770"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:21:01.739168 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:01.739137    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48b60203-3cb9-4133-813a-30a47a224770-maas-api-xks-ca" pod "48b60203-3cb9-4133-813a-30a47a224770" (UID: "48b60203-3cb9-4133-813a-30a47a224770"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:21:01.739239 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:01.739205    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b60203-3cb9-4133-813a-30a47a224770-kube-api-access-l728h" pod "48b60203-3cb9-4133-813a-30a47a224770" (UID: "48b60203-3cb9-4133-813a-30a47a224770"). InnerVolumeSpecName "kube-api-access-l728h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:21:01.838374 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:01.838258    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l728h\" (UniqueName: \"kubernetes.io/projected/48b60203-3cb9-4133-813a-30a47a224770-kube-api-access-l728h\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:01.838374 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:01.838312    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b60203-3cb9-4133-813a-30a47a224770-maas-api-xks-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:01.838374 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:01.838323    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b60203-3cb9-4133-813a-30a47a224770-maas-api-openshift-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:02.541182 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:02.541151    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"
Nov 02 21:21:02.541369 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:02.541152    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg" event={"ID":"48b60203-3cb9-4133-813a-30a47a224770","Type":"ContainerDied","Data":"d3690653ed9a9c4382b9fad08990487bf83c3b357823caac03524b3f4de68d0e"}
Nov 02 21:21:02.541369 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:02.541271    5224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3690653ed9a9c4382b9fad08990487bf83c3b357823caac03524b3f4de68d0e"
Nov 02 21:21:09.733674 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.733638    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"]
Nov 02 21:21:09.734174 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.733997    5224 cpu_manager.go:414] "containerMap: removing container" podUID="48b60203-3cb9-4133-813a-30a47a224770" containerName="revoke-keys"
Nov 02 21:21:09.734174 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.734009    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b60203-3cb9-4133-813a-30a47a224770" containerName="revoke-keys"
Nov 02 21:21:09.734174 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.734093    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b60203-3cb9-4133-813a-30a47a224770" containerName="revoke-keys"
Nov 02 21:21:09.737471 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.737453    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.791007 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.790978    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-phngj\""
Nov 02 21:21:09.806501 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.806472    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.806638 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.806507    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.806638 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.806524    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk4z6\" (UniqueName: \"kubernetes.io/projected/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-kube-api-access-bk4z6\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.806763 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.806635    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.806763 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.806688    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.806911 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.806812    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.806911 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.806850    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.807034 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.806926    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.807034 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.806960    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.844090 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.844062    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"]
Nov 02 21:21:09.907896 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.907859    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.908092 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.907903    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.908092 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.907919    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk4z6\" (UniqueName: \"kubernetes.io/projected/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-kube-api-access-bk4z6\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.908092 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.907951    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.908092 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.907990    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.908092 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.908022    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.908092 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.908049    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.908092 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.908081    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.908472 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.908109    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.908472 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.908339    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.908632 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.908593    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.908632 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.908627    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.908815 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.908674    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.908815 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.908703    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.910235 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.910213    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.910443 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.910425    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.965854 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.965825    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:09.966109 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:09.966088    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk4z6\" (UniqueName: \"kubernetes.io/projected/ec3c5974-99f9-4dce-90dd-98cd49e07c0c-kube-api-access-bk4z6\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jt7zx\" (UID: \"ec3c5974-99f9-4dce-90dd-98cd49e07c0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:10.049715 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:10.049686    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:10.465147 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:10.465063    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"]
Nov 02 21:21:10.497204 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:21:10.497177    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3c5974_99f9_4dce_90dd_98cd49e07c0c.slice/crio-e2eebbb5defb0e26d182babddc19a950b6173529c7289d795becfcf2d6e25181 WatchSource:0}: Error finding container e2eebbb5defb0e26d182babddc19a950b6173529c7289d795becfcf2d6e25181: Status 404 returned error can't find the container with id e2eebbb5defb0e26d182babddc19a950b6173529c7289d795becfcf2d6e25181
Nov 02 21:21:10.499234 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:10.499200    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:10.499338 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:10.499285    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:10.499403 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:10.499357    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:10.579279 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:10.579245    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" event={"ID":"ec3c5974-99f9-4dce-90dd-98cd49e07c0c","Type":"ContainerStarted","Data":"9b27b356307ef05325823232beb7ef7fa8eb63116c6fb6173e9e4c7a6ad695f8"}
Nov 02 21:21:10.579451 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:10.579300    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" event={"ID":"ec3c5974-99f9-4dce-90dd-98cd49e07c0c","Type":"ContainerStarted","Data":"e2eebbb5defb0e26d182babddc19a950b6173529c7289d795becfcf2d6e25181"}
Nov 02 21:21:10.700354 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:10.700300    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podStartSLOduration=1.7002848739999998 podStartE2EDuration="1.700284874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:10.698742115 +0000 UTC m=+5551.696430725" watchObservedRunningTime="2026-11-02 21:21:10.700284874 +0000 UTC m=+5551.697973482"
Nov 02 21:21:11.064791 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:11.058084    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:11.074405 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:11.073943    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 21:21:11.074405 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:11.074026    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 21:21:11.275584 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:11.275545    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:21:12.067792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:12.053170    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 21:21:12.067792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:12.053233    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 21:21:13.069793 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:13.064238    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 21:21:13.069793 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:13.064323    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 21:21:14.061292 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:14.061249    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 21:21:14.061618 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:14.061582    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 21:21:15.069800 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:15.059200    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 21:21:15.069800 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:15.059280    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 21:21:15.918679 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:15.918642    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:21:16.053792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:16.053113    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 21:21:16.053792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:16.053186    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 21:21:16.551315 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:16.551281    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:21:17.050815 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:17.050759    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 21:21:17.051026 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:17.050832    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 21:21:19.051527 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:19.051481    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:21:19.052087 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:19.051549    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:21:20.091975 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:20.091924    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:21:20.092474 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:20.092004    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:21:20.369441 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:20.369351    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:21:21.093133 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:21.093087    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:21:21.093636 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:21.093161    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:21:22.134947 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:22.134906    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:21:22.135357 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:22.134978    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:21:22.612317 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:22.612271    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:22.612702 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:22.612677    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:22.613451 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:22.613430    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx"
Nov 02 21:21:23.665809 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:23.665759    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:21:27.358281 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:21:27.358239    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:21:29.228365 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:29.228327    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:21:34.862816 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:34.862749    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:21:37.422253 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:37.422210    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:21:39.975048 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:39.975013    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"]
Nov 02 21:21:39.978211 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:39.978191    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:39.996150 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:39.996120    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:39.996299 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:39.996163    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:39.996299 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:39.996241    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:39.996299 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:39.996285    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:39.996475 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:39.996386    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:39.996475 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:39.996423    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:39.996475 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:39.996446    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:39.996638 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:39.996472    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:39.996638 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:39.996545    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czlf4\" (UniqueName: \"kubernetes.io/projected/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-kube-api-access-czlf4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.084420 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.084387    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"]
Nov 02 21:21:40.097384 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.097356    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.097577 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.097390    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.097577 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.097411    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.097577 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.097427    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.097577 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.097457    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czlf4\" (UniqueName: \"kubernetes.io/projected/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-kube-api-access-czlf4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.097577 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.097488    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.097577 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.097526    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.097577 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.097568    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.098000 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.097624    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.098000 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.097882    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.098000 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.097986    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.098155 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.098047    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.098155 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.098098    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.098294 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.098271    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.099974 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.099950    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.100179 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.100161    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.155763 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.155723    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.155954 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.155814    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czlf4\" (UniqueName: \"kubernetes.io/projected/9f1fd8b0-1637-4caa-b9cd-91675d97cf0d-kube-api-access-czlf4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-87hf6\" (UID: \"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.291127 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.291091    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:40.702465 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:21:40.702416    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1fd8b0_1637_4caa_b9cd_91675d97cf0d.slice/crio-eb5c7bdc2a6b9e3e81526f822ba363dca9031aa7744ef1edffe65662c12c8e2e WatchSource:0}: Error finding container eb5c7bdc2a6b9e3e81526f822ba363dca9031aa7744ef1edffe65662c12c8e2e: Status 404 returned error can't find the container with id eb5c7bdc2a6b9e3e81526f822ba363dca9031aa7744ef1edffe65662c12c8e2e
Nov 02 21:21:40.704713 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.704674    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:40.704849 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.704757    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:40.704849 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.704816    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:40.705965 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.705943    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"]
Nov 02 21:21:40.729117 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:40.729068    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6" event={"ID":"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d","Type":"ContainerStarted","Data":"eb5c7bdc2a6b9e3e81526f822ba363dca9031aa7744ef1edffe65662c12c8e2e"}
Nov 02 21:21:41.145061 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:41.145015    5224 scope.go:122] "RemoveContainer" containerID="ad2cde5d7a1ad234463cae92068b87fd27445aa9525f3ff8bbabd5975620d7b1"
Nov 02 21:21:41.735540 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:41.735492    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6" event={"ID":"9f1fd8b0-1637-4caa-b9cd-91675d97cf0d","Type":"ContainerStarted","Data":"e4385c097265230c8acbf646efd2f05dfccb7a0205702b62ee12d54d8ae6cc76"}
Nov 02 21:21:41.906003 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:41.905939    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6" podStartSLOduration=2.9059199749999998 podStartE2EDuration="2.905919975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:41.904302479 +0000 UTC m=+5582.901991098" watchObservedRunningTime="2026-11-02 21:21:41.905919975 +0000 UTC m=+5582.903608580"
Nov 02 21:21:42.292245 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:42.292203    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:42.293817 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:42.293794    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-87hf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 02 21:21:42.293903 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:42.293840    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6" podUID="9f1fd8b0-1637-4caa-b9cd-91675d97cf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 02 21:21:43.292546 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:43.292509    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-87hf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 02 21:21:43.292990 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:43.292568    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6" podUID="9f1fd8b0-1637-4caa-b9cd-91675d97cf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 02 21:21:44.291591 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:44.291552    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-87hf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 02 21:21:44.291801 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:44.291608    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6" podUID="9f1fd8b0-1637-4caa-b9cd-91675d97cf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 02 21:21:45.296845 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:45.296802    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-87hf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 02 21:21:45.297258 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:45.296870    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6" podUID="9f1fd8b0-1637-4caa-b9cd-91675d97cf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 02 21:21:46.292504 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:46.292463    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-87hf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 02 21:21:46.292694 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:46.292520    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6" podUID="9f1fd8b0-1637-4caa-b9cd-91675d97cf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 02 21:21:47.292538 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:47.292498    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-87hf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 02 21:21:47.293112 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:47.292567    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6" podUID="9f1fd8b0-1637-4caa-b9cd-91675d97cf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 02 21:21:49.292872 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:49.292758    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-87hf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:21:49.292872 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:49.292839    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6" podUID="9f1fd8b0-1637-4caa-b9cd-91675d97cf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:21:50.333969 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:50.333921    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-87hf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:21:50.334483 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:50.333987    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6" podUID="9f1fd8b0-1637-4caa-b9cd-91675d97cf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:21:50.978289 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:50.978251    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:50.978798 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:50.978733    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:50.979490 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:21:50.979474    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6"
Nov 02 21:21:57.357215 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:21:57.357170    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:22:10.230898 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.230864    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"]
Nov 02 21:22:10.234572 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.234547    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.265629 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.265597    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:22:10.365846 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.365805    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.366017 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.365922    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.366017 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.365953    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.366123 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.366051    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.366187 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.366130    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.366251 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.366212    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.366311 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.366262    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.366375 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.366321    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25hwt\" (UniqueName: \"kubernetes.io/projected/192d5786-0e48-4e66-8b9b-25b2e22e36c5-kube-api-access-25hwt\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.366375 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.366352    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.434221 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.434187    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"]
Nov 02 21:22:10.467435 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.467397    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.467615 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.467446    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.467615 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.467468    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.467615 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.467487    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25hwt\" (UniqueName: \"kubernetes.io/projected/192d5786-0e48-4e66-8b9b-25b2e22e36c5-kube-api-access-25hwt\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.467615 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.467505    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.467615 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.467537    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.467615 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.467571    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.467615 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.467597    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.468087 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.467637    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.468087 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.467958    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.468087 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.468062    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.468273 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.468070    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.468273 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.468138    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.468387 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.468339    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.469940 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.469918    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.470071 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.470052    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.524998 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.524959    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192d5786-0e48-4e66-8b9b-25b2e22e36c5-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.525181 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.525102    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25hwt\" (UniqueName: \"kubernetes.io/projected/192d5786-0e48-4e66-8b9b-25b2e22e36c5-kube-api-access-25hwt\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xhccj\" (UID: \"192d5786-0e48-4e66-8b9b-25b2e22e36c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.548069 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.548040    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:10.962616 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.962538    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"]
Nov 02 21:22:10.995345 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:22:10.995315    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192d5786_0e48_4e66_8b9b_25b2e22e36c5.slice/crio-aa2e8a8a483ffb882667efd5ad215a7ac63b3f411cae112c24c4e705c0ad66a4 WatchSource:0}: Error finding container aa2e8a8a483ffb882667efd5ad215a7ac63b3f411cae112c24c4e705c0ad66a4: Status 404 returned error can't find the container with id aa2e8a8a483ffb882667efd5ad215a7ac63b3f411cae112c24c4e705c0ad66a4
Nov 02 21:22:10.997849 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.997813    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:22:10.997963 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.997887    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:22:10.997963 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:10.997916    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:22:11.913870 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:11.913828    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj" event={"ID":"192d5786-0e48-4e66-8b9b-25b2e22e36c5","Type":"ContainerStarted","Data":"1a5c093b8735e5d6b82feb8d3f0468f35142a67700d59ea8464e264001c99ae0"}
Nov 02 21:22:11.914392 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:11.913882    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj" event={"ID":"192d5786-0e48-4e66-8b9b-25b2e22e36c5","Type":"ContainerStarted","Data":"aa2e8a8a483ffb882667efd5ad215a7ac63b3f411cae112c24c4e705c0ad66a4"}
Nov 02 21:22:12.548438 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:12.548399    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:12.569792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:12.561934    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xhccj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 21:22:12.569792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:12.562050    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj" podUID="192d5786-0e48-4e66-8b9b-25b2e22e36c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 21:22:13.548629 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:13.548593    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xhccj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 21:22:13.549141 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:13.548677    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj" podUID="192d5786-0e48-4e66-8b9b-25b2e22e36c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 21:22:14.548386 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:14.548355    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xhccj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 21:22:14.548562 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:14.548417    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj" podUID="192d5786-0e48-4e66-8b9b-25b2e22e36c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 21:22:14.600164 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:14.600108    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj" podStartSLOduration=4.600094651 podStartE2EDuration="4.600094651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:12.088469016 +0000 UTC m=+5613.086157626" watchObservedRunningTime="2026-11-02 21:22:14.600094651 +0000 UTC m=+5615.597783255"
Nov 02 21:22:14.601232 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:14.601213    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:22:15.551869 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:15.548862    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xhccj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 21:22:15.551869 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:15.548928    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj" podUID="192d5786-0e48-4e66-8b9b-25b2e22e36c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 21:22:16.299599 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:16.299401    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:22:16.548524 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:16.548489    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xhccj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 21:22:16.548720 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:16.548588    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj" podUID="192d5786-0e48-4e66-8b9b-25b2e22e36c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 21:22:16.715017 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:16.714924    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:22:16.962834 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:16.962802    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:22:17.433068 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:17.433021    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:22:17.548623 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:17.548585    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xhccj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 21:22:17.548851 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:17.548648    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj" podUID="192d5786-0e48-4e66-8b9b-25b2e22e36c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 21:22:18.549149 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:18.549110    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xhccj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 21:22:18.549550 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:18.549175    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj" podUID="192d5786-0e48-4e66-8b9b-25b2e22e36c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 21:22:20.589999 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:20.589953    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xhccj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:22:20.590548 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:20.590046    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj" podUID="192d5786-0e48-4e66-8b9b-25b2e22e36c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:22:21.384001 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:21.383948    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:21.384481 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:21.384459    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:21.385383 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:21.385357    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj"
Nov 02 21:22:21.443111 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:21.443063    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:22:27.358020 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:22:27.357982    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:22:28.536205 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:22:28.536169    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:22:57.357280 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:22:57.357244    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:23:05.145309 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:05.145272    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:23:16.623039 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:16.622973    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:17.298945 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:17.298902    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:27.356322 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:23:27.356235    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:23:29.070512 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:29.070481    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:23:29.219617 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:29.219581    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:31.347814 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:31.347755    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:39.768735 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:39.768698    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:23:42.308552 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:42.308514    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:46.797703 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:46.797669    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:47.281975 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.281940    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"]
Nov 02 21:23:47.285392 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.285376    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.315408 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.315371    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-credential-socket\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.315584 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.315502    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-socket\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.315584 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.315561    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-envoy\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.315726 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.315588    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-token\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.315816 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.315740    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-data\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.315816 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.315804    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-podinfo\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.315943 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.315921    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdmp5\" (UniqueName: \"kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-kube-api-access-jdmp5\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.316009 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.315975    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e2aabd-8233-4abe-a231-b74b458d5fd2-istiod-ca-cert\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.316067 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.316033    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-certs\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.338717 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.338689    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-742833-openshift-default-dockercfg-7xx7n\""
Nov 02 21:23:47.391734 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.391697    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"]
Nov 02 21:23:47.413283 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.413235    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:47.417257 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.417231    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmp5\" (UniqueName: \"kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-kube-api-access-jdmp5\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.417396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.417279    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e2aabd-8233-4abe-a231-b74b458d5fd2-istiod-ca-cert\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.417396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.417311    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-certs\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.417396 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.417336    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-credential-socket\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.417570 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.417396    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-socket\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.417570 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.417454    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-envoy\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.417570 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.417551    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-token\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.417734 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.417617    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-data\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.417734 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.417649    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-podinfo\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.417875 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.417788    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-certs\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.417938 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.417913    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-credential-socket\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.418189 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.418136    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-socket\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.418189 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.418164    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-data\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.418350 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.418180    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e2aabd-8233-4abe-a231-b74b458d5fd2-istiod-ca-cert\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.419914 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.419889    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-envoy\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.420214 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.420194    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-podinfo\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.475812 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.475655    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-token\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.477822 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.477793    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmp5\" (UniqueName: \"kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-kube-api-access-jdmp5\") pod \"e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") " pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.597913 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.597811    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:47.674686 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:47.674647    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:48.011597 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:23:48.011550    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e2aabd_8233_4abe_a231_b74b458d5fd2.slice/crio-fb023bdaeff152eaff53a1e5b068fefb1c4a189e8606548c49c786271de195ed WatchSource:0}: Error finding container fb023bdaeff152eaff53a1e5b068fefb1c4a189e8606548c49c786271de195ed: Status 404 returned error can't find the container with id fb023bdaeff152eaff53a1e5b068fefb1c4a189e8606548c49c786271de195ed
Nov 02 21:23:48.013942 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:48.013910    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:23:48.014075 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:48.013984    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:23:48.014075 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:48.014025    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:23:48.014592 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:48.014572    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"]
Nov 02 21:23:48.451784 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:48.451729    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" event={"ID":"14e2aabd-8233-4abe-a231-b74b458d5fd2","Type":"ContainerStarted","Data":"e12cab1ae3f8b0ebac0a4ad4f69d05c463572370144a64a014258ffa815b73f2"}
Nov 02 21:23:48.451964 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:48.451803    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" event={"ID":"14e2aabd-8233-4abe-a231-b74b458d5fd2","Type":"ContainerStarted","Data":"fb023bdaeff152eaff53a1e5b068fefb1c4a189e8606548c49c786271de195ed"}
Nov 02 21:23:48.570192 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:48.570138    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" podStartSLOduration=1.5701236729999999 podStartE2EDuration="1.570123673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:48.568662829 +0000 UTC m=+5709.566351438" watchObservedRunningTime="2026-11-02 21:23:48.570123673 +0000 UTC m=+5709.567812282"
Nov 02 21:23:48.598363 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:48.598336    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:49.226464 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:49.226426    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:49.604061 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:49.604031    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:50.479406 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:50.479228    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:50.487234 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:50.486984    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:23:51.501792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:51.501649    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:51.716225 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:51.716184    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:51.961372 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:51.961337    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:54.664251 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:54.664217    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:54.843630 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:54.843594    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:57.266449 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.266415    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"]
Nov 02 21:23:57.269919 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.269902    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:57.324048 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.324022    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-742833-serving-cert\""
Nov 02 21:23:57.336006 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.335963    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djmn9\" (UniqueName: \"kubernetes.io/projected/7d6016be-d21a-4914-8865-080c07362f1f-kube-api-access-djmn9\") pod \"maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:57.336144 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.336029    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6016be-d21a-4914-8865-080c07362f1f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:57.336144 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.336086    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:57.336224 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.336196    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:57.360238 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:23:57.360198    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:23:57.375057 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.375033    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:23:57.375221 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.375100    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:23:57.375221 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.375138    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 21:23:57.375343 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.375233    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nltm4\""
Nov 02 21:23:57.377019 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.377000    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"]
Nov 02 21:23:57.437162 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.437136    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djmn9\" (UniqueName: \"kubernetes.io/projected/7d6016be-d21a-4914-8865-080c07362f1f-kube-api-access-djmn9\") pod \"maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:57.437329 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.437182    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6016be-d21a-4914-8865-080c07362f1f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:57.437329 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.437212    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:57.437459 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.437341    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:57.437518 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:23:57.437467    5224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-serving-cert: secret "maas-api-e2e-worker-single-main-742833-serving-cert" not found
Nov 02 21:23:57.437577 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:23:57.437557    5224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-maas-api-tls podName:7d6016be-d21a-4914-8865-080c07362f1f nodeName:}" failed. No retries permitted until 2026-11-02 21:23:57.937538359 +0000 UTC m=+5718.935226961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-maas-api-tls") pod "maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq" (UID: "7d6016be-d21a-4914-8865-080c07362f1f") : secret "maas-api-e2e-worker-single-main-742833-serving-cert" not found
Nov 02 21:23:57.437870 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.437850    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6016be-d21a-4914-8865-080c07362f1f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:57.439685 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.439663    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:57.496404 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.496373    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djmn9\" (UniqueName: \"kubernetes.io/projected/7d6016be-d21a-4914-8865-080c07362f1f-kube-api-access-djmn9\") pod \"maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:57.496844 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.496818    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:57.750504 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.750417    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:23:57.943180 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.943148    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:57.945526 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:57.945508    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:58.182564 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:58.182533    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:23:58.544762 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:23:58.544732    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6016be_d21a_4914_8865_080c07362f1f.slice/crio-0f82dbc3a7e6422567468a695db7f5d2ff0eb5e251ef3461413e22ee5df2d9ee WatchSource:0}: Error finding container 0f82dbc3a7e6422567468a695db7f5d2ff0eb5e251ef3461413e22ee5df2d9ee: Status 404 returned error can't find the container with id 0f82dbc3a7e6422567468a695db7f5d2ff0eb5e251ef3461413e22ee5df2d9ee
Nov 02 21:23:58.546107 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:58.546082    5224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 21:23:58.548249 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:58.548229    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"]
Nov 02 21:23:59.536075 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:23:59.536036    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq" event={"ID":"7d6016be-d21a-4914-8865-080c07362f1f","Type":"ContainerStarted","Data":"0f82dbc3a7e6422567468a695db7f5d2ff0eb5e251ef3461413e22ee5df2d9ee"}
Nov 02 21:24:00.542241 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:00.542200    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq" event={"ID":"7d6016be-d21a-4914-8865-080c07362f1f","Type":"ContainerStarted","Data":"801b817487cab8261fc7256642444e322624085fa92786c1b42d5f86d3c0b7ea"}
Nov 02 21:24:00.542644 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:00.542352    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:24:00.659718 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:00.659669    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq" podStartSLOduration=2.56735873 podStartE2EDuration="3.65965559s" totalImagesPullingTime="1.09229686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:00.657514726 +0000 UTC m=+5721.655203335" watchObservedRunningTime="2026-11-02 21:24:00.65965559 +0000 UTC m=+5721.657344199"
Nov 02 21:24:02.133926 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:02.133894    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:05.039346 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:05.039312    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:24:06.574322 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:06.574288    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:24:06.713720 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:06.713683    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:11.207871 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:11.206057    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:14.629917 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:14.629884    5224 generic.go:356] "Generic (PLEG): container finished" podID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerID="e12cab1ae3f8b0ebac0a4ad4f69d05c463572370144a64a014258ffa815b73f2" exitCode=0
Nov 02 21:24:14.630364 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:14.629963    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" event={"ID":"14e2aabd-8233-4abe-a231-b74b458d5fd2","Type":"ContainerDied","Data":"e12cab1ae3f8b0ebac0a4ad4f69d05c463572370144a64a014258ffa815b73f2"}
Nov 02 21:24:14.630364 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:14.630336    5224 scope.go:122] "RemoveContainer" containerID="e12cab1ae3f8b0ebac0a4ad4f69d05c463572370144a64a014258ffa815b73f2"
Nov 02 21:24:14.631427 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:14.631403    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:24:14.631495 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:14.631469    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:24:14.631532 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:14.631502    5224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:24:15.636895 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:15.636859    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" event={"ID":"14e2aabd-8233-4abe-a231-b74b458d5fd2","Type":"ContainerStarted","Data":"333ae1c8428cebd371cff8b0205a4473ea83400a10a832dee2abefa2f534dc5f"}
Nov 02 21:24:16.074948 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:16.074891    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:16.268791 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:16.266523    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:16.598996 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:16.598962    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:24:16.600118 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:16.600091    5224 patch_prober.go:29] interesting pod/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 02 21:24:16.600242 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:16.600140    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 02 21:24:16.774532 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:16.774483    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:17.006368 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:17.006292    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:17.598436 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:17.598401    5224 patch_prober.go:29] interesting pod/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 02 21:24:17.598626 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:17.598454    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 02 21:24:17.626542 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:17.626495    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:18.598694 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:18.598660    5224 patch_prober.go:29] interesting pod/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 02 21:24:18.599116 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:18.598711    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 02 21:24:19.599107 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:19.599069    5224 patch_prober.go:29] interesting pod/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 02 21:24:19.599503 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:19.599128    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 02 21:24:20.478989 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:20.478949    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:24:20.598362 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:20.598310    5224 patch_prober.go:29] interesting pod/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 02 21:24:20.598552 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:20.598387    5224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 02 21:24:20.619348 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:20.619307    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:21.397054 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:21.397024    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:21.602477 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:21.602444    5224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:24:21.603593 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:21.603570    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:24:25.126258 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:25.126220    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:25.937071 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:25.937037    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:24:27.357526 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:24:27.357488    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:24:29.606591 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:29.606542    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:34.109825 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:34.109759    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:34.462643 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:34.462550    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:38.630101 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:38.630062    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:43.114591 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:43.114546    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:47.620663 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:47.620611    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:51.639235 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:51.639150    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:24:52.120288 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:52.119941    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:56.610166 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:24:56.610118    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:24:57.357548 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:24:57.357512    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:25:00.886026 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:00.885989    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:01.214574 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:01.214479    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:05.711462 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:05.711420    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:10.224868 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:10.224828    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:14.712748 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:14.712709    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:15.055948 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:15.055907    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"]
Nov 02 21:25:15.105892 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:15.105850    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zd572"]
Nov 02 21:25:15.166460 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:15.166425    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e66d2a8-994e-467c-ac11-47b3b34db7c6" path="/var/lib/kubelet/pods/1e66d2a8-994e-467c-ac11-47b3b34db7c6/volumes"
Nov 02 21:25:15.770352 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:15.770309    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:16.212870 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:16.212776    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:17.281058 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:17.281007    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:17.659152 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:17.659059    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:25:19.220505 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:19.220234    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:20.810559 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:20.810513    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:21.821517 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:21.821465    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:23.052288 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:23.052252    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:23.274243 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:23.274210    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:23.944538 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:23.944479    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:27.358919 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:25:27.358884    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:25:27.482737 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:27.482703    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:27.842406 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:27.842373    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"]
Nov 02 21:25:27.842608 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:27.842584    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq" podUID="7d6016be-d21a-4914-8865-080c07362f1f" containerName="maas-api" containerID="cri-o://801b817487cab8261fc7256642444e322624085fa92786c1b42d5f86d3c0b7ea" gracePeriod=30
Nov 02 21:25:27.982286 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:27.982243    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:29.027379 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.027329    5224 generic.go:356] "Generic (PLEG): container finished" podID="7d6016be-d21a-4914-8865-080c07362f1f" containerID="801b817487cab8261fc7256642444e322624085fa92786c1b42d5f86d3c0b7ea" exitCode=0
Nov 02 21:25:29.027798 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.027406    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq" event={"ID":"7d6016be-d21a-4914-8865-080c07362f1f","Type":"ContainerDied","Data":"801b817487cab8261fc7256642444e322624085fa92786c1b42d5f86d3c0b7ea"}
Nov 02 21:25:29.087956 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.087925    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:25:29.166609 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.166578    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6016be-d21a-4914-8865-080c07362f1f-kube-api-access-djmn9\" (UniqueName: \"kubernetes.io/projected/7d6016be-d21a-4914-8865-080c07362f1f-kube-api-access-djmn9\") pod \"7d6016be-d21a-4914-8865-080c07362f1f\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") "
Nov 02 21:25:29.166844 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.166643    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-metrics-certs\") pod \"7d6016be-d21a-4914-8865-080c07362f1f\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") "
Nov 02 21:25:29.166844 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.166685    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-maas-api-tls\") pod \"7d6016be-d21a-4914-8865-080c07362f1f\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") "
Nov 02 21:25:29.166844 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.166784    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6016be-d21a-4914-8865-080c07362f1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6016be-d21a-4914-8865-080c07362f1f-openshift-service-ca\") pod \"7d6016be-d21a-4914-8865-080c07362f1f\" (UID: \"7d6016be-d21a-4914-8865-080c07362f1f\") "
Nov 02 21:25:29.167159 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.167130    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6016be-d21a-4914-8865-080c07362f1f-openshift-service-ca" pod "7d6016be-d21a-4914-8865-080c07362f1f" (UID: "7d6016be-d21a-4914-8865-080c07362f1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:25:29.168920 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.168888    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-metrics-certs" pod "7d6016be-d21a-4914-8865-080c07362f1f" (UID: "7d6016be-d21a-4914-8865-080c07362f1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:25:29.169035 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.168967    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-maas-api-tls" pod "7d6016be-d21a-4914-8865-080c07362f1f" (UID: "7d6016be-d21a-4914-8865-080c07362f1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:25:29.169035 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.169013    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6016be-d21a-4914-8865-080c07362f1f-kube-api-access-djmn9" pod "7d6016be-d21a-4914-8865-080c07362f1f" (UID: "7d6016be-d21a-4914-8865-080c07362f1f"). InnerVolumeSpecName "kube-api-access-djmn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:25:29.267633 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.267597    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:25:29.267633 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.267631    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6016be-d21a-4914-8865-080c07362f1f-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:25:29.267891 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.267641    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6016be-d21a-4914-8865-080c07362f1f-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:25:29.267891 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:29.267652    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djmn9\" (UniqueName: \"kubernetes.io/projected/7d6016be-d21a-4914-8865-080c07362f1f-kube-api-access-djmn9\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:25:30.034158 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:30.034119    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq" event={"ID":"7d6016be-d21a-4914-8865-080c07362f1f","Type":"ContainerDied","Data":"0f82dbc3a7e6422567468a695db7f5d2ff0eb5e251ef3461413e22ee5df2d9ee"}
Nov 02 21:25:30.034158 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:30.034139    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"
Nov 02 21:25:30.034722 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:30.034190    5224 scope.go:122] "RemoveContainer" containerID="801b817487cab8261fc7256642444e322624085fa92786c1b42d5f86d3c0b7ea"
Nov 02 21:25:30.199797 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:30.199743    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"]
Nov 02 21:25:30.251131 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:30.251097    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-742833-859fcd6697-sr4mq"]
Nov 02 21:25:31.166619 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:31.166584    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6016be-d21a-4914-8865-080c07362f1f" path="/var/lib/kubelet/pods/7d6016be-d21a-4914-8865-080c07362f1f/volumes"
Nov 02 21:25:32.472891 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:32.472855    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:25:33.288571 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:33.288534    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:33.512711 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:33.512671    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:57.358167 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:25:57.358127    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:25:57.662011 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:57.661914    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:57.847259 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:57.847208    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:59.637271 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:59.637232    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:25:59.881855 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:25:59.881816    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:26:00.702075 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:00.701987    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:26:01.062470 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:01.062441    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:26:01.528258 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:01.528197    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"]
Nov 02 21:26:01.528924 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:01.528850    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerName="istio-proxy" containerID="cri-o://333ae1c8428cebd371cff8b0205a4473ea83400a10a832dee2abefa2f534dc5f" gracePeriod=30
Nov 02 21:26:01.609546 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:01.609510    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:26:02.046849 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:02.046812    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"]
Nov 02 21:26:02.052046 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:02.052022    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-bj7zg"]
Nov 02 21:26:03.166386 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:03.166354    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b60203-3cb9-4133-813a-30a47a224770" path="/var/lib/kubelet/pods/48b60203-3cb9-4133-813a-30a47a224770/volumes"
Nov 02 21:26:03.277848 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:03.277814    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:26:03.521150 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:03.521115    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:26:06.604367 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.604336    5224 patch_prober.go:29] interesting pod/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 02 21:26:06.604742 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.604401    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 02 21:26:06.770265 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.770240    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:26:06.932703 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.932613    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-certs\") pod \"14e2aabd-8233-4abe-a231-b74b458d5fd2\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") "
Nov 02 21:26:06.932928 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.932701    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-envoy\") pod \"14e2aabd-8233-4abe-a231-b74b458d5fd2\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") "
Nov 02 21:26:06.932928 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.932729    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-data\") pod \"14e2aabd-8233-4abe-a231-b74b458d5fd2\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") "
Nov 02 21:26:06.932928 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.932763    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-token\" (UniqueName: \"kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-token\") pod \"14e2aabd-8233-4abe-a231-b74b458d5fd2\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") "
Nov 02 21:26:06.932928 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.932875    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-podinfo\") pod \"14e2aabd-8233-4abe-a231-b74b458d5fd2\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") "
Nov 02 21:26:06.932928 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.932905    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-credential-socket\") pod \"14e2aabd-8233-4abe-a231-b74b458d5fd2\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") "
Nov 02 21:26:06.933208 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.932935    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e2aabd-8233-4abe-a231-b74b458d5fd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e2aabd-8233-4abe-a231-b74b458d5fd2-istiod-ca-cert\") pod \"14e2aabd-8233-4abe-a231-b74b458d5fd2\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") "
Nov 02 21:26:06.933208 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.932970    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-certs" pod "14e2aabd-8233-4abe-a231-b74b458d5fd2" (UID: "14e2aabd-8233-4abe-a231-b74b458d5fd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:26:06.933208 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.932982    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-kube-api-access-jdmp5\" (UniqueName: \"kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-kube-api-access-jdmp5\") pod \"14e2aabd-8233-4abe-a231-b74b458d5fd2\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") "
Nov 02 21:26:06.933208 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.933109    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-socket\") pod \"14e2aabd-8233-4abe-a231-b74b458d5fd2\" (UID: \"14e2aabd-8233-4abe-a231-b74b458d5fd2\") "
Nov 02 21:26:06.933208 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.933172    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-data" pod "14e2aabd-8233-4abe-a231-b74b458d5fd2" (UID: "14e2aabd-8233-4abe-a231-b74b458d5fd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:26:06.933496 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.933425    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-credential-socket" pod "14e2aabd-8233-4abe-a231-b74b458d5fd2" (UID: "14e2aabd-8233-4abe-a231-b74b458d5fd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:26:06.933496 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.933427    5224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:06.933496 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.933456    5224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-data\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:06.933678 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.933648    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-socket" pod "14e2aabd-8233-4abe-a231-b74b458d5fd2" (UID: "14e2aabd-8233-4abe-a231-b74b458d5fd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:26:06.933966 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.933935    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e2aabd-8233-4abe-a231-b74b458d5fd2-istiod-ca-cert" pod "14e2aabd-8233-4abe-a231-b74b458d5fd2" (UID: "14e2aabd-8233-4abe-a231-b74b458d5fd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:26:06.935714 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.935686    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-podinfo" pod "14e2aabd-8233-4abe-a231-b74b458d5fd2" (UID: "14e2aabd-8233-4abe-a231-b74b458d5fd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:26:06.935948 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.935899    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-envoy" pod "14e2aabd-8233-4abe-a231-b74b458d5fd2" (UID: "14e2aabd-8233-4abe-a231-b74b458d5fd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:26:06.936274 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.936249    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-token" pod "14e2aabd-8233-4abe-a231-b74b458d5fd2" (UID: "14e2aabd-8233-4abe-a231-b74b458d5fd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:26:06.936367 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:06.936353    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-kube-api-access-jdmp5" pod "14e2aabd-8233-4abe-a231-b74b458d5fd2" (UID: "14e2aabd-8233-4abe-a231-b74b458d5fd2"). InnerVolumeSpecName "kube-api-access-jdmp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:26:07.034840 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.034804    5224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-token\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:07.034840 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.034832    5224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-podinfo\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:07.034840 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.034842    5224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-credential-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:07.034840 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.034851    5224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e2aabd-8233-4abe-a231-b74b458d5fd2-istiod-ca-cert\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:07.035120 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.034860    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdmp5\" (UniqueName: \"kubernetes.io/projected/14e2aabd-8233-4abe-a231-b74b458d5fd2-kube-api-access-jdmp5\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:07.035120 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.034870    5224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-workload-socket\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:07.035120 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.034878    5224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e2aabd-8233-4abe-a231-b74b458d5fd2-istio-envoy\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:07.219484 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.219400    5224 generic.go:356] "Generic (PLEG): container finished" podID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerID="333ae1c8428cebd371cff8b0205a4473ea83400a10a832dee2abefa2f534dc5f" exitCode=0
Nov 02 21:26:07.219484 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.219447    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"
Nov 02 21:26:07.219484 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.219474    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" event={"ID":"14e2aabd-8233-4abe-a231-b74b458d5fd2","Type":"ContainerDied","Data":"333ae1c8428cebd371cff8b0205a4473ea83400a10a832dee2abefa2f534dc5f"}
Nov 02 21:26:07.219730 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.219527    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9" event={"ID":"14e2aabd-8233-4abe-a231-b74b458d5fd2","Type":"ContainerDied","Data":"fb023bdaeff152eaff53a1e5b068fefb1c4a189e8606548c49c786271de195ed"}
Nov 02 21:26:07.219730 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.219550    5224 scope.go:122] "RemoveContainer" containerID="333ae1c8428cebd371cff8b0205a4473ea83400a10a832dee2abefa2f534dc5f"
Nov 02 21:26:07.228401 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.228383    5224 scope.go:122] "RemoveContainer" containerID="e12cab1ae3f8b0ebac0a4ad4f69d05c463572370144a64a014258ffa815b73f2"
Nov 02 21:26:07.236037 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.236020    5224 scope.go:122] "RemoveContainer" containerID="333ae1c8428cebd371cff8b0205a4473ea83400a10a832dee2abefa2f534dc5f"
Nov 02 21:26:07.236303 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:26:07.236287    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333ae1c8428cebd371cff8b0205a4473ea83400a10a832dee2abefa2f534dc5f\": container with ID starting with 333ae1c8428cebd371cff8b0205a4473ea83400a10a832dee2abefa2f534dc5f not found: ID does not exist" containerID="333ae1c8428cebd371cff8b0205a4473ea83400a10a832dee2abefa2f534dc5f"
Nov 02 21:26:07.236373 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.236314    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333ae1c8428cebd371cff8b0205a4473ea83400a10a832dee2abefa2f534dc5f"} err="failed to get container status \"333ae1c8428cebd371cff8b0205a4473ea83400a10a832dee2abefa2f534dc5f\": rpc error: code = NotFound desc = could not find container \"333ae1c8428cebd371cff8b0205a4473ea83400a10a832dee2abefa2f534dc5f\": container with ID starting with 333ae1c8428cebd371cff8b0205a4473ea83400a10a832dee2abefa2f534dc5f not found: ID does not exist"
Nov 02 21:26:07.236373 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.236330    5224 scope.go:122] "RemoveContainer" containerID="e12cab1ae3f8b0ebac0a4ad4f69d05c463572370144a64a014258ffa815b73f2"
Nov 02 21:26:07.236596 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:26:07.236573    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12cab1ae3f8b0ebac0a4ad4f69d05c463572370144a64a014258ffa815b73f2\": container with ID starting with e12cab1ae3f8b0ebac0a4ad4f69d05c463572370144a64a014258ffa815b73f2 not found: ID does not exist" containerID="e12cab1ae3f8b0ebac0a4ad4f69d05c463572370144a64a014258ffa815b73f2"
Nov 02 21:26:07.236698 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.236600    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12cab1ae3f8b0ebac0a4ad4f69d05c463572370144a64a014258ffa815b73f2"} err="failed to get container status \"e12cab1ae3f8b0ebac0a4ad4f69d05c463572370144a64a014258ffa815b73f2\": rpc error: code = NotFound desc = could not find container \"e12cab1ae3f8b0ebac0a4ad4f69d05c463572370144a64a014258ffa815b73f2\": container with ID starting with e12cab1ae3f8b0ebac0a4ad4f69d05c463572370144a64a014258ffa815b73f2 not found: ID does not exist"
Nov 02 21:26:07.386751 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.386714    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"]
Nov 02 21:26:07.438019 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:07.437982    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-742833-openshift-default-759ffccb89j6tl9"]
Nov 02 21:26:09.166134 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:09.166098    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" path="/var/lib/kubelet/pods/14e2aabd-8233-4abe-a231-b74b458d5fd2/volumes"
Nov 02 21:26:17.615071 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:17.615030    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:26:18.312543 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:18.312458    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:26:18.501202 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:18.501139    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:26:22.775025 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:22.774983    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:26:27.358675 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:26:27.358631    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:26:41.211953 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:41.211923    5224 scope.go:122] "RemoveContainer" containerID="5e8bae27d7e45cf7eb0cb6fb5406dd09f8e41a54bbbaa862404fb4db41ef2df7"
Nov 02 21:26:45.440738 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:45.440705    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:26:49.868248 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:26:49.868213    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:26:57.358496 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:26:57.358454    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:27:17.963880 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:27:17.963840    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:27:18.660618 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:27:18.660582    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:27:20.967078 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:27:20.967041    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:27:23.665268 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:27:23.664870    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:27:23.665268 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:27:23.664968    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:27:27.356912 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:27:27.356877    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:27:32.663716 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:27:32.663676    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:27:35.404482 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:27:35.404434    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:27:36.175020 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:27:36.174981    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:27:38.659912 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:27:38.659870    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:27:38.660555 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:27:38.660520    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:27:41.235887 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:27:41.235849    5224 scope.go:122] "RemoveContainer" containerID="fa6402353d3490a392ed5aa5f6e0af1e6cfc9d4e5cf3379587506959a7cc13f8"
Nov 02 21:27:47.473480 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:27:47.473440    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:27:56.939680 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:27:56.939588    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:27:57.361137 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:27:57.361085    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:28:03.942344 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:28:03.942308    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:28:04.177859 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:28:04.177825    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:28:14.935694 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:28:14.935658    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:28:18.436636 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:28:18.436593    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:28:27.358278 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:28:27.358238    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:28:57.357525 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:28:57.357490    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:29:02.860929 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:29:02.860891    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:29:04.797001 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:29:04.796965    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:29:05.826376 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:29:05.826334    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:29:15.246055 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:29:15.246022    5224 generic.go:356] "Generic (PLEG): container finished" podID="69235044-5904-4cad-93db-315e6e36f8fd" containerID="cf5f0add92e0490c31c5786b0b4c52f8de3464c60bdab8279291f7f013468442" exitCode=1
Nov 02 21:29:15.246597 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:29:15.246091    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k" event={"ID":"69235044-5904-4cad-93db-315e6e36f8fd","Type":"ContainerDied","Data":"cf5f0add92e0490c31c5786b0b4c52f8de3464c60bdab8279291f7f013468442"}
Nov 02 21:29:15.246597 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:29:15.246449    5224 scope.go:122] "RemoveContainer" containerID="cf5f0add92e0490c31c5786b0b4c52f8de3464c60bdab8279291f7f013468442"
Nov 02 21:29:16.253578 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:29:16.253542    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k" event={"ID":"69235044-5904-4cad-93db-315e6e36f8fd","Type":"ContainerStarted","Data":"74d13b60048bc1712a2601678439049bde1548b88e9b7b3c08580d989467cdd1"}
Nov 02 21:29:16.254016 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:29:16.253797    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k"
Nov 02 21:29:18.751359 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:29:18.751275    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:29:19.328959 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:29:19.328926    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:29:27.260927 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:29:27.260894    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rkw4k"
Nov 02 21:29:27.358216 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:29:27.358181    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:29:33.377327 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:29:33.377290    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:29:57.357545 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:29:57.357501    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:30:04.844219 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:30:04.844173    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:30:19.001949 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:30:19.001908    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:30:19.652785 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:30:19.652738    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:30:25.206110 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:30:25.206074    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:30:27.357635 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:30:27.357597    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:30:36.068287 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:30:36.068253    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:30:36.251001 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:30:36.250969    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:30:36.539302 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:30:36.539272    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:30:38.872083 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:30:38.872046    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:30:49.068859 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:30:49.068756    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:30:57.357282 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:30:57.357244    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:31:07.279523 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:31:07.279482    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:31:17.569873 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:31:17.569829    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:31:19.332053 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:31:19.332016    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:31:27.357531 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:31:27.357494    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:31:31.866326 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:31:31.866289    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:31:50.268618 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:31:50.268580    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:31:57.358570 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:31:57.358533    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:32:18.843214 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:32:18.843113    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:32:19.674155 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:32:19.674118    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:32:20.320792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:32:20.317853    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:32:27.358419 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:32:27.358378    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:32:29.463126 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:32:29.463085    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:32:41.335369 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:32:41.335337    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:32:57.357598 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:32:57.357561    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:33:04.055830 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:33:04.051906    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:33:13.063913 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:33:13.063878    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:33:20.020472 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:33:20.020441    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:33:20.661190 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:33:20.661151    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:33:21.073257 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:33:21.073222    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:33:27.357511 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:33:27.357476    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:33:29.856965 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:33:29.856932    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:33:38.953781 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:33:38.953742    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:33:50.746064 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:33:50.745980    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:33:57.356583 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:33:57.356548    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:33:58.935281 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:33:58.935231    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:34:13.956331 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:34:13.956301    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:34:20.448875 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:34:20.448826    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:34:22.027987 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:34:22.027944    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-87hf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:34:22.028435 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:34:22.028015    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6" podUID="9f1fd8b0-1637-4caa-b9cd-91675d97cf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:34:22.431055 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:34:22.430949    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xhccj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:34:22.431055 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:34:22.431018    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xhccj" podUID="192d5786-0e48-4e66-8b9b-25b2e22e36c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:34:24.373877 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:34:24.373836    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:34:27.356831 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:34:27.356796    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:34:34.702399 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:34:34.702368    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:34:37.565943 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:34:37.565908    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:34:47.273861 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:34:47.273829    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:34:57.357974 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:34:57.357934    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:34:58.537287 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:34:58.537254    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:35:17.640617 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:35:17.640578    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:35:20.746613 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:35:20.746504    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:35:21.330122 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:35:21.330078    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:35:27.361984 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:35:27.361942    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:35:28.370690 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:35:28.370651    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:35:48.660511 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:35:48.660475    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:35:50.027862 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:35:50.027821    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:35:50.740944 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:35:50.740897    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:35:51.161361 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:35:51.161278    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:35:55.174834 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:35:55.174796    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:35:55.416374 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:35:55.416343    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:35:57.358592 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:35:57.358555    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:35:58.321781 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:35:58.321738    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:35:58.492452 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:35:58.492411    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:00.993020 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:00.992946    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:01.214214 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:01.214178    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:01.395494 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:01.395453    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:01.833783 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:01.833731    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:36:05.630328 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:05.630286    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:10.246456 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:10.246416    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:13.582448 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:13.582414    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:36:14.736179 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:14.736139    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:19.526573 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:19.526537    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:19.905037 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:19.904936    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:20.480249 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:20.480209    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:20.663633 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:20.663589    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:21.110279 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:21.110238    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:21.661688 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:21.661653    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:24.237925 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:24.237885    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:24.646232 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:24.646190    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:27.357398 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:36:27.357358    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:36:28.738289 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:28.738253    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:28.884756 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:28.884720    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:36:33.205150 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:33.205095    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:37.735567 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:37.735531    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:42.211579 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:42.211539    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:46.711045 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:46.711003    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:51.217359 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:51.217258    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:55.531254 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:55.531219    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:36:55.722083 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:55.722050    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:56.258623 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:36:56.258585    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:36:57.358445 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:36:57.358395    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:37:00.223445 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:00.223403    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:04.720609 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:04.720570    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:09.139007 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:09.138784    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:13.623869 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:13.623827    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:18.116145 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:18.116055    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:21.410715 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:21.410676    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:21.587470 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:21.587434    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:21.985897 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:21.985860    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:22.611682 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:22.611637    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:24.563424 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:24.563393    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:37:26.550010 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:26.549972    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:27.107553 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:27.107525    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:27.363055 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:37:27.362982    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:37:27.461543 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:27.461490    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:27.696109 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:27.696025    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:27.969213 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:27.969134    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:31.285354 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:31.285316    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:33.384590 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:33.384552    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:33.613876 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:33.613841    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:37:35.030948 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:35.030912    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:37:57.356505 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:37:57.356472    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:37:57.495333 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:37:57.495293    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:01.739643 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:01.739608    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:01.911564 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:01.911524    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:02.925557 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:02.925521    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:03.151189 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:03.151144    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:03.583988 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:03.583947    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:03.767055 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:03.767017    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:04.710800 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:04.710747    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:05.062021 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:05.061980    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:05.649651 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:05.649605    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:09.220272 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:09.220230    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:09.458390 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:09.458353    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:11.299081 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:11.299050    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:21.740580 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:21.740501    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:38:21.761712 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:21.761675    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:22.329891 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:22.329852    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:23.618821 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:23.618656    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:38:23.618821 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:23.618740    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:38:27.357112 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:38:27.357077    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:38:41.961904 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:41.961866    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:53.884483 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:38:53.884444    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:38:57.357755 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:38:57.357720    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:39:22.422390 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:39:22.422343    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:39:22.694828 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:39:22.694716    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:39:27.356778 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:39:27.356725    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:39:50.022843 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:39:50.022752    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:39:53.670792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:39:53.666039    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:39:53.670792 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:39:53.666116    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jt7zx" podUID="ec3c5974-99f9-4dce-90dd-98cd49e07c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:39:55.974060 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:39:55.974020    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:39:57.357115 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:39:57.357078    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:40:08.556621 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:08.556586    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:40:18.057340 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:18.057301    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:40:22.364293 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:22.364222    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:40:26.068978 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:26.068939    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:40:27.358343 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:40:27.358306    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:40:29.652308 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.652269    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"]
Nov 02 21:40:29.652714 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.652639    5224 cpu_manager.go:414] "containerMap: removing container" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerName="istio-proxy"
Nov 02 21:40:29.652714 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.652650    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerName="istio-proxy"
Nov 02 21:40:29.652714 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.652678    5224 cpu_manager.go:414] "containerMap: removing container" podUID="7d6016be-d21a-4914-8865-080c07362f1f" containerName="maas-api"
Nov 02 21:40:29.652714 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.652684    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6016be-d21a-4914-8865-080c07362f1f" containerName="maas-api"
Nov 02 21:40:29.652876 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.652748    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerName="istio-proxy"
Nov 02 21:40:29.652876 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.652761    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerName="istio-proxy"
Nov 02 21:40:29.652876 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.652780    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6016be-d21a-4914-8865-080c07362f1f" containerName="maas-api"
Nov 02 21:40:29.655801 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.655763    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:29.747088 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.747048    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-maas-api-tls\") pod \"maas-api-794d57d6f9-48gsl\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:29.747342 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.747109    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-metrics-certs\") pod \"maas-api-794d57d6f9-48gsl\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:29.747342 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.747286    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9jr7\" (UniqueName: \"kubernetes.io/projected/0613600c-4d06-4bcd-9441-ab9d228c58be-kube-api-access-k9jr7\") pod \"maas-api-794d57d6f9-48gsl\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:29.747342 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.747335    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0613600c-4d06-4bcd-9441-ab9d228c58be-openshift-service-ca\") pod \"maas-api-794d57d6f9-48gsl\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:29.760709 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.760674    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:40:29.760709 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.760674    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:40:29.760709 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.760714    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nltm4\""
Nov 02 21:40:29.761054 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.760783    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 21:40:29.761054 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.760826    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 21:40:29.762546 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.762522    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"]
Nov 02 21:40:29.848224 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.848182    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jr7\" (UniqueName: \"kubernetes.io/projected/0613600c-4d06-4bcd-9441-ab9d228c58be-kube-api-access-k9jr7\") pod \"maas-api-794d57d6f9-48gsl\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:29.848224 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.848229    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0613600c-4d06-4bcd-9441-ab9d228c58be-openshift-service-ca\") pod \"maas-api-794d57d6f9-48gsl\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:29.848514 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.848282    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-maas-api-tls\") pod \"maas-api-794d57d6f9-48gsl\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:29.848514 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.848306    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-metrics-certs\") pod \"maas-api-794d57d6f9-48gsl\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:29.849002 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.848979    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0613600c-4d06-4bcd-9441-ab9d228c58be-openshift-service-ca\") pod \"maas-api-794d57d6f9-48gsl\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:29.850757 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.850731    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-metrics-certs\") pod \"maas-api-794d57d6f9-48gsl\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:29.850908 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.850888    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-maas-api-tls\") pod \"maas-api-794d57d6f9-48gsl\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:29.906257 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.906182    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jr7\" (UniqueName: \"kubernetes.io/projected/0613600c-4d06-4bcd-9441-ab9d228c58be-kube-api-access-k9jr7\") pod \"maas-api-794d57d6f9-48gsl\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:29.971454 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:29.971416    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:30.350926 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:40:30.350893    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0613600c_4d06_4bcd_9441_ab9d228c58be.slice/crio-f1e0a515c3c29aec404e15952ec543daf8b4e8c9cfc999223a5e8d0aa94447ea WatchSource:0}: Error finding container f1e0a515c3c29aec404e15952ec543daf8b4e8c9cfc999223a5e8d0aa94447ea: Status 404 returned error can't find the container with id f1e0a515c3c29aec404e15952ec543daf8b4e8c9cfc999223a5e8d0aa94447ea
Nov 02 21:40:30.352449 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:30.352427    5224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 21:40:30.354332 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:30.354299    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"]
Nov 02 21:40:30.739733 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:30.739645    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl" event={"ID":"0613600c-4d06-4bcd-9441-ab9d228c58be","Type":"ContainerStarted","Data":"f1e0a515c3c29aec404e15952ec543daf8b4e8c9cfc999223a5e8d0aa94447ea"}
Nov 02 21:40:31.746080 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:31.745986    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl" event={"ID":"0613600c-4d06-4bcd-9441-ab9d228c58be","Type":"ContainerStarted","Data":"4a648448910a149814acd356ee07f0d0c2ba3b9c02714e52f2f0330c573af012"}
Nov 02 21:40:31.746551 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:31.746136    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:31.862621 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:31.862569    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl" podStartSLOduration=1.94103284 podStartE2EDuration="2.862555189s" totalImagesPullingTime="921.522349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:31.86209998 +0000 UTC m=+6712.859788589" watchObservedRunningTime="2026-11-02 21:40:31.862555189 +0000 UTC m=+6712.860243879"
Nov 02 21:40:37.757406 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:37.757368    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:37.763840 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:37.763800    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"]
Nov 02 21:40:37.774521 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:37.774485    5224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl" podUID="0613600c-4d06-4bcd-9441-ab9d228c58be" containerName="maas-api" containerID="cri-o://4a648448910a149814acd356ee07f0d0c2ba3b9c02714e52f2f0330c573af012" gracePeriod=30
Nov 02 21:40:37.814906 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:37.814870    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54b974b655-xk98q"]
Nov 02 21:40:37.815314 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:37.815298    5224 cpu_manager.go:414] "containerMap: removing container" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerName="istio-proxy"
Nov 02 21:40:37.815367 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:37.815315    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e2aabd-8233-4abe-a231-b74b458d5fd2" containerName="istio-proxy"
Nov 02 21:40:37.818175 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:37.818159    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:37.894064 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:37.894027    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b974b655-xk98q"]
Nov 02 21:40:37.929994 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:37.929961    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/345c21ca-b8e3-4088-8745-8f498bc52f4f-openshift-service-ca\") pod \"maas-api-54b974b655-xk98q\" (UID: \"345c21ca-b8e3-4088-8745-8f498bc52f4f\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:37.930208 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:37.930030    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6p4g\" (UniqueName: \"kubernetes.io/projected/345c21ca-b8e3-4088-8745-8f498bc52f4f-kube-api-access-z6p4g\") pod \"maas-api-54b974b655-xk98q\" (UID: \"345c21ca-b8e3-4088-8745-8f498bc52f4f\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:37.930208 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:37.930122    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345c21ca-b8e3-4088-8745-8f498bc52f4f-maas-api-tls\") pod \"maas-api-54b974b655-xk98q\" (UID: \"345c21ca-b8e3-4088-8745-8f498bc52f4f\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:37.930322 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:37.930224    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/345c21ca-b8e3-4088-8745-8f498bc52f4f-metrics-certs\") pod \"maas-api-54b974b655-xk98q\" (UID: \"345c21ca-b8e3-4088-8745-8f498bc52f4f\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:38.023450 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.023424    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:38.031394 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.031365    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/345c21ca-b8e3-4088-8745-8f498bc52f4f-metrics-certs\") pod \"maas-api-54b974b655-xk98q\" (UID: \"345c21ca-b8e3-4088-8745-8f498bc52f4f\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:38.031571 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.031413    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/345c21ca-b8e3-4088-8745-8f498bc52f4f-openshift-service-ca\") pod \"maas-api-54b974b655-xk98q\" (UID: \"345c21ca-b8e3-4088-8745-8f498bc52f4f\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:38.031571 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.031468    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6p4g\" (UniqueName: \"kubernetes.io/projected/345c21ca-b8e3-4088-8745-8f498bc52f4f-kube-api-access-z6p4g\") pod \"maas-api-54b974b655-xk98q\" (UID: \"345c21ca-b8e3-4088-8745-8f498bc52f4f\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:38.031571 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.031488    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345c21ca-b8e3-4088-8745-8f498bc52f4f-maas-api-tls\") pod \"maas-api-54b974b655-xk98q\" (UID: \"345c21ca-b8e3-4088-8745-8f498bc52f4f\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:38.032268 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.032243    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/345c21ca-b8e3-4088-8745-8f498bc52f4f-openshift-service-ca\") pod \"maas-api-54b974b655-xk98q\" (UID: \"345c21ca-b8e3-4088-8745-8f498bc52f4f\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:38.034035 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.034007    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/345c21ca-b8e3-4088-8745-8f498bc52f4f-metrics-certs\") pod \"maas-api-54b974b655-xk98q\" (UID: \"345c21ca-b8e3-4088-8745-8f498bc52f4f\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:38.034137 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.034050    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345c21ca-b8e3-4088-8745-8f498bc52f4f-maas-api-tls\") pod \"maas-api-54b974b655-xk98q\" (UID: \"345c21ca-b8e3-4088-8745-8f498bc52f4f\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:38.089933 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.089904    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6p4g\" (UniqueName: \"kubernetes.io/projected/345c21ca-b8e3-4088-8745-8f498bc52f4f-kube-api-access-z6p4g\") pod \"maas-api-54b974b655-xk98q\" (UID: \"345c21ca-b8e3-4088-8745-8f498bc52f4f\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:38.132083 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.132041    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0613600c-4d06-4bcd-9441-ab9d228c58be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0613600c-4d06-4bcd-9441-ab9d228c58be-openshift-service-ca\") pod \"0613600c-4d06-4bcd-9441-ab9d228c58be\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") "
Nov 02 21:40:38.132083 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.132089    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-metrics-certs\") pod \"0613600c-4d06-4bcd-9441-ab9d228c58be\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") "
Nov 02 21:40:38.132318 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.132207    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0613600c-4d06-4bcd-9441-ab9d228c58be-kube-api-access-k9jr7\" (UniqueName: \"kubernetes.io/projected/0613600c-4d06-4bcd-9441-ab9d228c58be-kube-api-access-k9jr7\") pod \"0613600c-4d06-4bcd-9441-ab9d228c58be\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") "
Nov 02 21:40:38.132318 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.132279    5224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-maas-api-tls\") pod \"0613600c-4d06-4bcd-9441-ab9d228c58be\" (UID: \"0613600c-4d06-4bcd-9441-ab9d228c58be\") "
Nov 02 21:40:38.133838 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.132666    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0613600c-4d06-4bcd-9441-ab9d228c58be-openshift-service-ca" pod "0613600c-4d06-4bcd-9441-ab9d228c58be" (UID: "0613600c-4d06-4bcd-9441-ab9d228c58be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:40:38.135400 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.135366    5224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0613600c-4d06-4bcd-9441-ab9d228c58be-openshift-service-ca\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:38.140544 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.140497    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-maas-api-tls" pod "0613600c-4d06-4bcd-9441-ab9d228c58be" (UID: "0613600c-4d06-4bcd-9441-ab9d228c58be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:40:38.140682 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.140540    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0613600c-4d06-4bcd-9441-ab9d228c58be-kube-api-access-k9jr7" pod "0613600c-4d06-4bcd-9441-ab9d228c58be" (UID: "0613600c-4d06-4bcd-9441-ab9d228c58be"). InnerVolumeSpecName "kube-api-access-k9jr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:40:38.140682 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.140510    5224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-metrics-certs" pod "0613600c-4d06-4bcd-9441-ab9d228c58be" (UID: "0613600c-4d06-4bcd-9441-ab9d228c58be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:40:38.164901 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.164868    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:38.236640 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.236613    5224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9jr7\" (UniqueName: \"kubernetes.io/projected/0613600c-4d06-4bcd-9441-ab9d228c58be-kube-api-access-k9jr7\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:38.236850 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.236836    5224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-maas-api-tls\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:38.236915 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.236907    5224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0613600c-4d06-4bcd-9441-ab9d228c58be-metrics-certs\") on node \"ip-10-0-134-196.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:38.530268 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:40:38.530237    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345c21ca_b8e3_4088_8745_8f498bc52f4f.slice/crio-e2841c446db132bb214c383243e516f5725a8cc7c155cd706297e6e3acaa120e WatchSource:0}: Error finding container e2841c446db132bb214c383243e516f5725a8cc7c155cd706297e6e3acaa120e: Status 404 returned error can't find the container with id e2841c446db132bb214c383243e516f5725a8cc7c155cd706297e6e3acaa120e
Nov 02 21:40:38.533036 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.533013    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b974b655-xk98q"]
Nov 02 21:40:38.781495 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.781403    5224 generic.go:356] "Generic (PLEG): container finished" podID="0613600c-4d06-4bcd-9441-ab9d228c58be" containerID="4a648448910a149814acd356ee07f0d0c2ba3b9c02714e52f2f0330c573af012" exitCode=0
Nov 02 21:40:38.781495 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.781470    5224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"
Nov 02 21:40:38.782007 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.781487    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl" event={"ID":"0613600c-4d06-4bcd-9441-ab9d228c58be","Type":"ContainerDied","Data":"4a648448910a149814acd356ee07f0d0c2ba3b9c02714e52f2f0330c573af012"}
Nov 02 21:40:38.782007 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.781541    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl" event={"ID":"0613600c-4d06-4bcd-9441-ab9d228c58be","Type":"ContainerDied","Data":"f1e0a515c3c29aec404e15952ec543daf8b4e8c9cfc999223a5e8d0aa94447ea"}
Nov 02 21:40:38.782007 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.781571    5224 scope.go:122] "RemoveContainer" containerID="4a648448910a149814acd356ee07f0d0c2ba3b9c02714e52f2f0330c573af012"
Nov 02 21:40:38.782874 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.782848    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q" event={"ID":"345c21ca-b8e3-4088-8745-8f498bc52f4f","Type":"ContainerStarted","Data":"e2841c446db132bb214c383243e516f5725a8cc7c155cd706297e6e3acaa120e"}
Nov 02 21:40:38.799649 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.799617    5224 scope.go:122] "RemoveContainer" containerID="4a648448910a149814acd356ee07f0d0c2ba3b9c02714e52f2f0330c573af012"
Nov 02 21:40:38.799995 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:40:38.799975    5224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a648448910a149814acd356ee07f0d0c2ba3b9c02714e52f2f0330c573af012\": container with ID starting with 4a648448910a149814acd356ee07f0d0c2ba3b9c02714e52f2f0330c573af012 not found: ID does not exist" containerID="4a648448910a149814acd356ee07f0d0c2ba3b9c02714e52f2f0330c573af012"
Nov 02 21:40:38.800073 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.800008    5224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a648448910a149814acd356ee07f0d0c2ba3b9c02714e52f2f0330c573af012"} err="failed to get container status \"4a648448910a149814acd356ee07f0d0c2ba3b9c02714e52f2f0330c573af012\": rpc error: code = NotFound desc = could not find container \"4a648448910a149814acd356ee07f0d0c2ba3b9c02714e52f2f0330c573af012\": container with ID starting with 4a648448910a149814acd356ee07f0d0c2ba3b9c02714e52f2f0330c573af012 not found: ID does not exist"
Nov 02 21:40:38.949130 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.949097    5224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"]
Nov 02 21:40:38.953151 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:38.953121    5224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-794d57d6f9-48gsl"]
Nov 02 21:40:39.166631 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:39.166546    5224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0613600c-4d06-4bcd-9441-ab9d228c58be" path="/var/lib/kubelet/pods/0613600c-4d06-4bcd-9441-ab9d228c58be/volumes"
Nov 02 21:40:39.794828 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:39.794790    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q" event={"ID":"345c21ca-b8e3-4088-8745-8f498bc52f4f","Type":"ContainerStarted","Data":"05a2d299490f647b8eef9017b08e7e4baf01b0fe52c84d58a01bbcbccba21283"}
Nov 02 21:40:39.795317 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:39.794924    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:39.913089 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:39.913034    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q" podStartSLOduration=2.067361457 podStartE2EDuration="2.91301946s" totalImagesPullingTime="845.658003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:39.911830135 +0000 UTC m=+6720.909518738" watchObservedRunningTime="2026-11-02 21:40:39.91301946 +0000 UTC m=+6720.910708069"
Nov 02 21:40:42.241484 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:42.241449    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:40:45.804935 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:40:45.804906    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54b974b655-xk98q"
Nov 02 21:40:57.357759 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:40:57.357711    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:41:18.886884 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:41:18.886793    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:41:24.703178 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:41:24.703136    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:41:24.885311 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:41:24.885274    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:41:27.425028 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:41:27.424958    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:41:30.777633 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:41:30.777592    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:41:52.259850 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:41:52.259817    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:41:57.358259 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:41:57.358221    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:42:04.171326 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:42:04.171285    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:42:07.026638 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:42:07.026578    5224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-87hf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:42:07.027220 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:42:07.026684    5224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-87hf6" podUID="9f1fd8b0-1637-4caa-b9cd-91675d97cf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:42:23.096358 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:42:23.096320    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:42:23.714165 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:42:23.714128    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:42:27.358119 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:42:27.358077    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:42:34.733030 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:42:34.732998    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:42:45.471726 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:42:45.471692    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:42:57.356953 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:42:57.356877    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:42:57.549158 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:42:57.549125    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:43:06.756887 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:43:06.756852    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:43:23.445573 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:43:23.445529    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:43:27.356853 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:43:27.356816    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:43:28.706919 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:43:28.706884    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:43:29.948244 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:43:29.948210    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:43:40.670332 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:43:40.670288    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:43:49.131977 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:43:49.131922    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:43:49.311802 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:43:49.311749    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:43:49.874195 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:43:49.874159    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:43:57.358056 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:43:57.358000    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:44:10.362629 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:44:10.362593    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:44:19.677092 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:44:19.677010    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:44:23.839691 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:44:23.839655    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:44:24.363001 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:44:24.362964    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:44:27.356815 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:44:27.356760    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:44:27.629380 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:44:27.629293    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:44:57.358061 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:44:57.358019    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:45:24.150378 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:45:24.150342    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:45:26.267140 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:45:26.267102    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:45:27.358915 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:45:27.358876    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:45:51.843157 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:45:51.843066    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:45:57.422286 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:45:57.422214    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:46:24.483797 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:46:24.483749    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:46:27.066174 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:46:27.066132    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2ptm5"]
Nov 02 21:46:27.358406 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:46:27.358315    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:46:56.232570 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:46:56.232533    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2vzvt_cb2304f9-c44a-44a4-a0b5-6029ffd68eec/manager/0.log"
Nov 02 21:46:56.769109 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:46:56.769073    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8z8gw_e49ef931-b6e2-4394-80f3-881252323f43/manager/0.log"
Nov 02 21:46:57.358628 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:46:57.358590    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:46:59.279999 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:46:59.279967    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-54b974b655-xk98q_345c21ca-b8e3-4088-8745-8f498bc52f4f/maas-api/0.log"
Nov 02 21:47:06.792895 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:06.792857    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2ptm5_1f554e67-b005-4058-9728-e047ee0f014b/limitador/0.log"
Nov 02 21:47:07.011787 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:07.011734    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rkw4k_69235044-5904-4cad-93db-315e6e36f8fd/manager/1.log"
Nov 02 21:47:07.783456 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:07.783421    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rkw4k_69235044-5904-4cad-93db-315e6e36f8fd/manager/0.log"
Nov 02 21:47:09.202492 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:09.202458    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c9c5d7d8f-g4db8_a3e59fe7-4223-4124-ba42-c5dde81b4c72/kube-auth-proxy/0.log"
Nov 02 21:47:10.324560 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:10.324528    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-87hf6_9f1fd8b0-1637-4caa-b9cd-91675d97cf0d/istio-proxy/0.log"
Nov 02 21:47:10.931540 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:10.931511    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-jt7zx_ec3c5974-99f9-4dce-90dd-98cd49e07c0c/istio-proxy/0.log"
Nov 02 21:47:11.845954 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:11.845923    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-xhccj_192d5786-0e48-4e66-8b9b-25b2e22e36c5/istio-proxy/0.log"
Nov 02 21:47:12.449737 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:12.449704    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7454465567-f226n_3e02684d-c911-4625-ade1-6b04120058d8/istio-proxy/0.log"
Nov 02 21:47:24.816143 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:24.816093    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:47:25.340053 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:25.340010    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:47:25.898072 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:25.898035    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:47:26.130464 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:26.130428    5224 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log" limit=1024
Nov 02 21:47:27.356359 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:47:27.356318    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:47:36.704328 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:36.704287    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kf87v_cd0732dd-9b73-4204-a64b-ba95c2aa6007/global-pull-secret-syncer/0.log"
Nov 02 21:47:38.677890 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:38.677861    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jwvp7_086392f7-3abd-4c9b-ae7b-69ccd1985d1b/konnectivity-agent/0.log"
Nov 02 21:47:39.723343 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:39.723304    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2vzvt_cb2304f9-c44a-44a4-a0b5-6029ffd68eec/manager/0.log"
Nov 02 21:47:40.917892 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:40.917852    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8z8gw_e49ef931-b6e2-4394-80f3-881252323f43/manager/0.log"
Nov 02 21:47:41.050406 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:41.050379    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-196.us-west-2.compute.internal_09b897c9011dced250aef3fd51f6bd3f/haproxy/0.log"
Nov 02 21:47:47.154675 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.154639    5224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"]
Nov 02 21:47:47.155083 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.155007    5224 cpu_manager.go:414] "containerMap: removing container" podUID="0613600c-4d06-4bcd-9441-ab9d228c58be" containerName="maas-api"
Nov 02 21:47:47.155083 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.155018    5224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0613600c-4d06-4bcd-9441-ab9d228c58be" containerName="maas-api"
Nov 02 21:47:47.155157 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.155099    5224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0613600c-4d06-4bcd-9441-ab9d228c58be" containerName="maas-api"
Nov 02 21:47:47.158260 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.158241    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.212857 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.212833    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k9pw5\"/\"kube-root-ca.crt\""
Nov 02 21:47:47.264122 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.264084    5224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k9pw5\"/\"openshift-service-ca.crt\""
Nov 02 21:47:47.264301 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.264178    5224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k9pw5\"/\"default-dockercfg-sn7x5\""
Nov 02 21:47:47.265673 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.265651    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"]
Nov 02 21:47:47.349621 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.349586    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-podres\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.349820 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.349637    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-lib-modules\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.349820 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.349800    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-proc\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.349903 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.349834    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-sys\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.349950 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.349935    5224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhptz\" (UniqueName: \"kubernetes.io/projected/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-kube-api-access-xhptz\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.450407 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.450322    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhptz\" (UniqueName: \"kubernetes.io/projected/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-kube-api-access-xhptz\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.450407 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.450377    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-podres\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.450407 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.450406    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-lib-modules\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.450654 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.450448    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-proc\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.450654 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.450465    5224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-sys\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.450654 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.450543    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-podres\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.450654 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.450554    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-proc\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.450654 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.450556    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-lib-modules\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.450654 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.450562    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-sys\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.508075 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.508038    5224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhptz\" (UniqueName: \"kubernetes.io/projected/b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9-kube-api-access-xhptz\") pod \"perf-node-gather-daemonset-xfbwt\" (UID: \"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:47.770978 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:47.770953    5224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:48.006462 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:48.006431    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2vzvt_cb2304f9-c44a-44a4-a0b5-6029ffd68eec/manager/0.log"
Nov 02 21:47:48.134340 ip-10-0-134-196 kubenswrapper[5224]: W1102 21:47:48.134314    5224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb7aef9a1_e25c_4de4_8d41_2de6bcbe2ba9.slice/crio-b8b710d96fa34acb2f9964daf0ecd36a169497b60d6a9db8e78f52fbc98d70d8 WatchSource:0}: Error finding container b8b710d96fa34acb2f9964daf0ecd36a169497b60d6a9db8e78f52fbc98d70d8: Status 404 returned error can't find the container with id b8b710d96fa34acb2f9964daf0ecd36a169497b60d6a9db8e78f52fbc98d70d8
Nov 02 21:47:48.137066 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:48.137038    5224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"]
Nov 02 21:47:48.683890 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:48.683862    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2ptm5_1f554e67-b005-4058-9728-e047ee0f014b/limitador/0.log"
Nov 02 21:47:49.115315 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:49.115187    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt" event={"ID":"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9","Type":"ContainerStarted","Data":"7dba44294e38354f7713860c7a10e661cbce3f3f41e47ac0b50a75f553453caa"}
Nov 02 21:47:49.115515 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:49.115322    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:49.115515 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:49.115354    5224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt" event={"ID":"b7aef9a1-e25c-4de4-8d41-2de6bcbe2ba9","Type":"ContainerStarted","Data":"b8b710d96fa34acb2f9964daf0ecd36a169497b60d6a9db8e78f52fbc98d70d8"}
Nov 02 21:47:49.239433 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:49.239371    5224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt" podStartSLOduration=2.239353216 podStartE2EDuration="2.239353216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:49.237738793 +0000 UTC m=+7150.235427427" watchObservedRunningTime="2026-11-02 21:47:49.239353216 +0000 UTC m=+7150.237041825"
Nov 02 21:47:49.292895 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:49.292857    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rkw4k_69235044-5904-4cad-93db-315e6e36f8fd/manager/1.log"
Nov 02 21:47:49.312793 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:49.312735    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8z8gw_e49ef931-b6e2-4394-80f3-881252323f43/manager/0.log"
Nov 02 21:47:49.362904 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:49.362868    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rkw4k_69235044-5904-4cad-93db-315e6e36f8fd/manager/0.log"
Nov 02 21:47:52.076322 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:52.076282    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18474534-444d-45ea-8a53-20cace7aa3e1/alertmanager/0.log"
Nov 02 21:47:52.484881 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:52.484793    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18474534-444d-45ea-8a53-20cace7aa3e1/config-reloader/0.log"
Nov 02 21:47:52.892326 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:52.892299    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18474534-444d-45ea-8a53-20cace7aa3e1/kube-rbac-proxy-web/0.log"
Nov 02 21:47:53.300297 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:53.300267    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18474534-444d-45ea-8a53-20cace7aa3e1/kube-rbac-proxy/0.log"
Nov 02 21:47:53.721656 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:53.721556    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18474534-444d-45ea-8a53-20cace7aa3e1/kube-rbac-proxy-metric/0.log"
Nov 02 21:47:54.129692 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:54.129665    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18474534-444d-45ea-8a53-20cace7aa3e1/prom-label-proxy/0.log"
Nov 02 21:47:54.538164 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:54.538126    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18474534-444d-45ea-8a53-20cace7aa3e1/init-config-reloader/0.log"
Nov 02 21:47:55.129261 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:55.129236    5224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-xfbwt"
Nov 02 21:47:55.423042 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:55.422962    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2vzvt_cb2304f9-c44a-44a4-a0b5-6029ffd68eec/manager/0.log"
Nov 02 21:47:56.721157 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:56.721123    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8z8gw_e49ef931-b6e2-4394-80f3-881252323f43/manager/0.log"
Nov 02 21:47:57.358056 ip-10-0-134-196 kubenswrapper[5224]: E1102 21:47:57.358016    5224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6kgdb" containerName="extractor"
Nov 02 21:47:58.859973 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:58.859934    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9zfw_cb5b4895-f27a-4de9-a084-4b6bf0a4af9d/node-exporter/0.log"
Nov 02 21:47:59.267552 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:59.267519    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9zfw_cb5b4895-f27a-4de9-a084-4b6bf0a4af9d/kube-rbac-proxy/0.log"
Nov 02 21:47:59.677326 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:47:59.677253    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9zfw_cb5b4895-f27a-4de9-a084-4b6bf0a4af9d/init-textfile/0.log"
Nov 02 21:48:02.576834 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:02.576801    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd453033-2746-4b42-aabc-c364430b8ce3/prometheus/0.log"
Nov 02 21:48:02.986442 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:02.986367    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd453033-2746-4b42-aabc-c364430b8ce3/config-reloader/0.log"
Nov 02 21:48:03.394720 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:03.394691    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd453033-2746-4b42-aabc-c364430b8ce3/thanos-sidecar/0.log"
Nov 02 21:48:03.954612 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:03.954584    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd453033-2746-4b42-aabc-c364430b8ce3/kube-rbac-proxy-web/0.log"
Nov 02 21:48:04.362909 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:04.362882    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd453033-2746-4b42-aabc-c364430b8ce3/kube-rbac-proxy/0.log"
Nov 02 21:48:04.771341 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:04.771306    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd453033-2746-4b42-aabc-c364430b8ce3/kube-rbac-proxy-thanos/0.log"
Nov 02 21:48:05.179668 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:05.179591    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd453033-2746-4b42-aabc-c364430b8ce3/init-config-reloader/0.log"
Nov 02 21:48:05.599383 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:05.599351    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-mfdwd_0ab2031f-ddc9-4e69-a860-4d326697de18/prometheus-operator/0.log"
Nov 02 21:48:05.999611 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:05.999535    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-mfdwd_0ab2031f-ddc9-4e69-a860-4d326697de18/kube-rbac-proxy/0.log"
Nov 02 21:48:06.411481 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:06.411450    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-lbcmb_e4c78d9b-2b7d-4522-af17-f595eae2ab56/prometheus-operator-admission-webhook/0.log"
Nov 02 21:48:06.829498 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:06.829460    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dcb85b9b6-7xp5g_92b5e340-4e43-45e1-85e3-a867fda1c9f6/telemeter-client/0.log"
Nov 02 21:48:07.237553 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:07.237478    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dcb85b9b6-7xp5g_92b5e340-4e43-45e1-85e3-a867fda1c9f6/reload/0.log"
Nov 02 21:48:07.646489 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:07.646462    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dcb85b9b6-7xp5g_92b5e340-4e43-45e1-85e3-a867fda1c9f6/kube-rbac-proxy/0.log"
Nov 02 21:48:08.865712 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:08.865682    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c9c5d7d8f-g4db8_a3e59fe7-4223-4124-ba42-c5dde81b4c72/kube-auth-proxy/0.log"
Nov 02 21:48:09.707354 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:09.707319    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-87hf6_9f1fd8b0-1637-4caa-b9cd-91675d97cf0d/istio-proxy/0.log"
Nov 02 21:48:10.126977 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:10.126950    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-jt7zx_ec3c5974-99f9-4dce-90dd-98cd49e07c0c/istio-proxy/0.log"
Nov 02 21:48:10.755935 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:10.755905    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-xhccj_192d5786-0e48-4e66-8b9b-25b2e22e36c5/istio-proxy/0.log"
Nov 02 21:48:11.172811 ip-10-0-134-196 kubenswrapper[5224]: I1102 21:48:11.172762    5224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7454465567-f226n_3e02684d-c911-4625-ade1-6b04120058d8/istio-proxy/0.log"